acceptodds
Under review as a conference paper at ICLR 2027

World4Act Policy: Learning Explicit 4D World Dynamics for Robot Policy Training

Abstract

World–Action Models improve robotic policy learning by coupling action prediction with forecasts of future visual dynamics. However, existing approaches predominantly supervise future evolution in RGB space, leaving the underlying 3D geometry and motion implicit. Although recent 4D world models incorporate geometric or motion cues, dense 3D structure and scene flow have yet to be systematically explored as action-conditioned prediction targets. Here we present World4Act Policy, an action-centred 4D World–Action Model that explicitly learns future visual appearance, 3D geometry and motion alongside robot actions. Given current multi-view RGB observations, robot states and language instructions, the model jointly predicts future action chunks, multi-view RGB frames, image-aligned point maps and 3D scene flow. To transfer pretrained video-generative priors to these 4D modalities, we develop dedicated geometry and motion variational autoencoders, together with a progressive training strategy that proceeds from 4D representation learning to world modelling and, ultimately, joint World–Action learning. For efficient deployment, we further introduce a modality-asymmetric causal architecture that decouples action generation from future visual and 4D tokens, such that all world-generation branches can be discarded at inference time. To provide the dense supervision required for learning action-conditioned 4D dynamics at scale, we also curate Robo4D_Dyn Dataset, a large-scale embodied-video dataset annotated with dense point maps and 3D scene flow. Experiments demonstrate that World4Act Policy predicts future 4D dynamics more accurately than representative 4D world models and achieves higher manipulation success rates than both video-based and 4D policy baselines, while preserving low-latency, action-only inference.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action model, 4d dynamic, manipulation task

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NuDSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

World4Act Policy jointly learns action generation and future RGB, point-map, and scene-flow prediction within an action-centered World–Action Model. It combines modality-specific VAEs, three-stage training, and the Robo4D_Dyn dataset, while using causal attention to remove future-world prediction at deployment. The results support the value of explicit 3D supervision for prediction and manipulation, with little additional action-inference latency.

Critical strengths
  1. The motivation is sound, and the representations are well chosen. Point maps and scene flow explicitly describe 3D positions and displacements while preserving image alignment, providing a natural interface for reusing video-generation priors.
  2. The training and deployment design is coherent. Specialized VAEs and progressive training accommodate different modalities, while causal attention makes future-world prediction optional at deployment. The reported action-inference latency is 0.37 s, close to GigaWorld-Policy’s 0.36 s.
  3. The ablations provide encouraging evidence. Table 3(a) examines geometry, motion, and joint supervision. The full model improves success by 9 percentage points over the internal RGB-only variant on both randomized RoboTwin and the real-world average, and by 3 points over Projective 4D.
Critical weaknesses
  1. Stronger action–dynamics coupling is not directly established. Table 1 evaluates Stage-II models, which are not action-conditioned, and therefore does not test whether different actions produce the correct predicted 3D outcomes. Although Stage III exposes the world branch to action representations, the paper provides neither controlled comparisons under different actions nor action-removal/shuffling ablations. The proposed coupling therefore remains insufficiently validated.
  2. The independent contribution of 3D scene flow remains unclear. The Projective-to-Direct comparison changes both geometry and motion targets, preventing attribution to scene flow alone. The RGB + Scene Flow ablation supports the value of motion supervision, but does not establish its advantage over 2D optical-flow supervision under otherwise matched conditions.
  3. The real-world evaluation needs clearer documentation. The four tasks in Table 2 lack a corresponding overview or clearly identified policy-execution sequences. Although the main text refers readers to the supplement, task-specific initial conditions and success criteria remain underspecified.
Questions for authors
  1. What are the spatial and temporal sampling settings for Stage I? Please provide the training resolution, sequence length, frame interval, and padding scheme for short sequences. Since scene-flow magnitude depends on the temporal interval, please also clarify how temporal sampling is handled across data sources.
  2. How accurately do the specialized VAEs reconstruct geometry and motion? Figure 6 is schematic, while Table 3(b) primarily reports downstream effects. Could the authors provide held-out reconstruction metrics against the unadapted RGB VAE, together with examples around motion boundaries and manipulated objects? Since the VAEs are subsequently frozen, their reconstruction fidelity warrants direct evaluation.
  3. Which data are used in Stage III, and how is training organized? Please specify the data sources, demonstrations per task, sampling proportions, and whether training is joint across tasks or task-specific. How are the clean and randomized RoboTwin data used?
3 comments
AuthorsOpenReview ↗

Response to Reviewer NuDS

We thank the reviewer for the detailed comments. We address the concerns below.

W1: Stronger action–dynamics coupling is not established: no comparisons under different actions, no action-removal or shuffling ablations.

Response to W1: Following the reviewer's suggestion, we ran these controlled comparisons and ablations on RoboTwin 2.0. They establish the stronger action–dynamics coupling directly: the predicted 3D motion follows the conditioning actions, degrades when they are shuffled or removed, and is coupled more tightly than with projective targets (Table S6).

  • Setup. From each of 100 held-out RoboTwin states we execute two action chunks in the simulator: A, the demonstrated one, and B, A with a 3–6 cm end-effector offset. World4Act (Stage III) is conditioned on a chunk, and EPE3D\mathrm{EPE}_{3D} is measured against simulator scene flow on dynamic regions.
  • Results. Conditioned on A, the prediction matches the outcome of A (0.118) and not that of B (0.241), and vice versa; it is closer to the outcome of its own chunk in 94% and 93% of the states. A chunk from another episode raises EPE3D\mathrm{EPE}_{3D} to 0.268, and removing the action tokens to 0.182.
  • Stronger coupling. The Projective 4D variant separates the two outcomes less (1.6× vs. 2.0×; 88% vs. 94%). Removing the attention from future-world to action tokens in Stage III (Eq. 7) lowers RoboTwin success from 0.90/0.90 to 0.87/0.86.

Table S6. Stage-III prediction under action interventions (EPE3D\mathrm{EPE}_{3D}, dynamic regions).

Model, actionsvs. outcome Avs. outcome BOwn closer (%)
World4Act, A0.1180.24194
World4Act, B0.2370.12493
World4Act, shuffled0.2680.271–
World4Act, removed0.1820.185–
Projective 4D, A0.1410.22988

W2: The advantage of 3D scene flow over 2D optical flow under matched conditions is not established.

Response to W2: With the geometry target fixed, 3D scene flow is ahead of 2D optical flow in all three settings (Table S7). Depth and optical flow are projections of the same 4RC labels, with VAEs adapted by the same Stage-I schedule. It lowers Stage-II EPE3D\mathrm{EPE}_{3D} by 7.5% with point maps and 6.0% with depth, and raises success by 1–2 points in every setting.

Table S7. Geometry and motion targets varied independently.

GeometryMotionSuccess (rand.) ↑
–2D optical flow0.82
–3D scene flow0.84
Depth2D optical flow0.87
Depth3D scene flow0.88
Point map2D optical flow0.88
Point map3D scene flow (ours)0.90

W3: Real-world initial conditions and success criteria are underspecified.

Response to W3: We apologize for the missing task definitions. All methods share the protocol below, with 100 recorded initial configurations per task. Clean the Desk: 5 of 12 objects in a 60×40 cm area; all in the storage box within 90 s. Scan a QR Code: package within ±10 cm and ±30° yaw; code decoded within 45 s. Sweep up Trash: 6–8 paper scraps; all in the dustpan within 90 s. Stack Bowls: 3 bowls of different sizes within ±12 cm; nested and stable for 3 s within 60 s.

Q1: Stage-I sampling settings, and temporal sampling across sources?

Response to Q1: Scene flow is always expressed as displacement per 0.1 s. Each view is resized to 480×640; a clip has 13 point maps and 12 forward flow maps. Each source uses the frame stride closest to 0.1 s, and the flow is rescaled from the actual interval to 0.1 s. Sources below 10 Hz (RT-1, Bridge) give no flow targets. Clips shorter than 13 frames are discarded, so no temporal padding is used.

Q2: How accurately do the adapted VAEs reconstruct geometry and motion?

Response to Q2: Stage I is a key part of our method. Many prior works, including our baseline RynnWorld-4D, keep the pretrained VAE frozen. We find that fine-tuning it gives much better geometry and flow reconstruction: on 2,000 held-out clips it removes 69–75% of the error (Table S8). MotionCrafter (CVPR 2026) likewise shows that fine-tuning the geometry and motion VAEs is necessary. Fig. 6 will show real boundary and object examples.

Table S8. Held-out VAE reconstruction error (flow: EPE3D\mathrm{EPE}_{3D}).

VAERelp\mathrm{Rel}_p ↓Flow, allFlow, boundariesFlow, objects
Unadapted RGB VAE0.0710.0520.1270.081
Adapted (ours)0.0180.0140.0390.022

Q3: Which data are used in Stage III, and how are clean and randomized RoboTwin data used?

Response to Q3: Stage III uses target-platform demonstrations only, with 4D targets from 4RC. Simulation: one multi-task policy for all 50 RoboTwin 2.0 tasks, trained on 50 clean and 500 randomized demonstrations per task, with uniform task sampling; one checkpoint is evaluated under both conditions. Real world: one policy for the four tasks, 100 demonstrations each.

AuthorsOpenReview ↗

Response to Reviewer NuDS

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

AuthorsOpenReview ↗

Further discussion with Reviewer NuDS

Dear Reviewer NuDS,

Thank you for your thoughtful feedback on our paper. As the discussion period nears its end, we would like to see whether our previous responses have addressed your questions.

If any points would benefit from further clarification, please let us know. We would be glad to provide additional details.

Best regards,

Authors

Reviewer_weNFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes World4Act Policy, a World Action Model that predicts future head-view point maps and 3D scene flow, in addition to future multi view RGB frames, as auxiliary supervision for learning robot actions. Built on Wan 2.2 and GigaWorld-Policy, the model uses separate VAEs for geometry and motion. Training proceeds in three stages: VAE adaptation, 4D world-model pretraining, and joint world–action learning. A modality-asymmetric causal mask prevents action tokens from attending to future-world tokens, so the 4D branch can be skipped at deployment. The 4D supervision comes from a 31.8M-frame dataset built from public robot datasets and annotated with 4RC reconstructions. The paper reports better 4D prediction than RynnWorld-4D, higher success rates than VLA and WAM baselines on RoboTwin 2.0 and four real-world tasks, and an action-only latency of 0.370 s.

Critical strengths
  1. Direct 4D prediction targets for a World–Action Model. The paper makes future point maps and 3D scene flow native prediction targets of the policy model, instead of supervising only in RGB space or recovering 3D motion from depth and optical flow. Dedicated geometry and motion VAEs let these targets reuse a pretrained video-generation backbone
  2. 4D supervision without extra deployment cost. With the action-centered causal design, the 4D prediction acts only as training-time supervision and can be skipped at inference. The policy keeps the latency of an RGB-based WAM, whereas the prior 4D policy still runs its predictive backbone at inference.
Critical weaknesses
  1. The point maps and scene flow are 4RC reconstructions, not measured geometry. Table 1 evaluates on the same kind of annotations, which World4Act is trained on and the baselines are not. The comparison therefore favors World4Act, and it measures agreement with 4RC rather than physical accuracy, as the authors themselves note. No evaluation against measured geometry is reported, such as sensor depth or the simulator geometry in RoboTwin, so the claim of more accurate 4D prediction is not yet well supported.
  2. The paper argues that explicit 3D geometry and motion should help tasks that need accurate 3D localization or relative placement, but I could not find results that show such a benefit. RGB-only methods, including VLAs such as Qwen-RobotManip, WAMs such as OpenWAM, and even WAMs without pretraining such as FastWAM, already reach performance comparable to World4Act on RoboTwin 2.0. The reported 0.90 therefore does not by itself show an advantage of the 3D representation. The paper provides neither a task-level analysis nor tasks that specifically require 3D reasoning, so it remains unclear where the 4D targets actually help.
  3. Constructing Robo4D Dyn requires running 4RC reconstruction, CoTracker3-based checks and Qwen3-VL captioning over 31.8M frames, but the paper reports neither the cost of this pipeline nor how many clips are filtered out. For training, Table 4 lists only learning rates, update steps and batch sizes, without the hardware or total compute of the three stages. Even the hardware used for the latency comparison in Fig. 5 is not stated. Without these numbers, it is hard to judge whether the 4D pipeline is practical to scale or to reproduce, and how its cost compares with that of RGB-only WAMs.
Questions for authors
  1. Where do the 4D targets actually make a difference? The introduction motivates the method with grasping and placement, which need accurate 3D localization and relative geometry. In 4.2, the gains on Sweep up Trash and Scan a QR Code are attributed to this, but the differences are small. On RoboTwin 2.0, RGB-only VLAs and WAMs, already reach a level similar to World4Act. Is there a type of task where an RGB-only WAM consistently fails and the 4D supervision is clearly what makes World4Act succeed?
  2. Is the 4D pipeline worth its cost? Compared with an RGB-only WAM, the method needs 4RC reconstruction and filtering over the whole dataset, two extra VAEs, and a separate 4D pretraining stage. None of these costs is reported. Given that the real-robot average goes from 83.0% to 85.0% over GigaWorld-Policy, I am not sure the same effort spent on more RGB or action data would not bring a similar gain. How do you see the trade-off between adding 4D supervision and simply scaling RGB-only training?
  3. What is the role of the full 4D rollout? At deployment only the action stream is used, while the full rollout takes 4.08 s. So the 4D branch mainly serves as training-time supervision. The paper says the rollout "remains available when explicit prediction is required", but does not show any use for it. If the 4D branch is essentially an auxiliary loss, why does it need to be a full generative 4D model rather than a lighter 3D prediction objective?
3 comments
AuthorsOpenReview ↗

Response to Reviewer weNF

We thank the reviewer and address each point below.

W1: Table 1 is evaluated on 4RC annotations; no evaluation against measured geometry.

Response to W1: We agree that Table 1 measures agreement with 4RC. Following the reviewer's suggestion, we evaluated against RoboTwin simulator geometry, and World4Act remains ahead (Table S3).

  • Why 4RC annotations. No real-world manipulation dataset provides dense 3D geometry and scene flow, so on real data this supervision can only come from an annotation model. Simulators offer exact geometry, but their RGB differs markedly from real images, so we keep them out of Robo4D_Dyn. Robo4D_Dyn is itself a core contribution that we hope to provide to the community.
  • RoboTwin simulator geometry. We evaluate the Stage-II checkpoints of Table 1, without fine-tuning, on held-out RoboTwin 2.0 clips (not in our training data) with simulator point maps and scene flow. World4Act lowers Relp\mathrm{Rel}_p by 12.9% and EPE3D\mathrm{EPE}_{3D} by 6.4% relative to RynnWorld-4D and stays ahead of our projective variant trained on the same data. The 4RC labels themselves are close to the simulator (last row).

Table S3. Stage-II prediction against RoboTwin simulator ground truth.

MethodRelp\mathrm{Rel}_p ↓EPE3D\mathrm{EPE}_{3D} ↓
RynnWorld-4D0.3410.156
Projective 4D (same data)0.3120.161
World4Act0.2970.146
4RC annotations0.0410.027

W2 and Q1: RGB-only methods are comparable on RoboTwin 2.0; where do the 4D targets actually help?

Response to W2 and Q1: We agree that the RoboTwin 2.0 average of differently pretrained methods does not isolate 3D ability. During the rebuttal we added two real-robot tasks that require 3D perception, on which World4Act outperforms all four RGB-only methods by 20–30 points on average (Table S4).

  • Tasks. Reach past Vase: grasp a cup on the far side of the table without touching a vase in the way. Retrieve from Cabinet: open a cabinet and take out the object inside. All methods are fine-tuned on the same 100 demonstrations per task and evaluated for 50 trials per task.
  • Results. World4Act succeeds in 80% of the trials, against 50–60% for the RGB-only methods (p<0.01p < 0.01 against the best of them), including GigaWorld-Policy (52%), which shares our backbone.

Table S4. Real-world success (%) on two 3D-demanding tasks (50 trials each).

MethodReach past VaseRetrieve from CabinetAverage
Qwen-RobotManip524850
OpenWAM645660
FastWAM545856
GigaWorld-Policy564852
World4Act827880

W3 and Q2: Costs and hardware are not reported; would scaling RGB or action data give a similar gain?

Response to W3 and Q2: The 4D pipeline costs 2.8× the compute of the RGB-only variant, and spending this budget on RGB-only training does not close the gap (Table S5).

  • Costs. Annotation is a one-time pass over 40.0M candidate frames: 1,000 A800 GPU-hours; the checks reject 20.5%, leaving 31.8M. Training on 8 A800 GPUs takes 180/900/440 GPU-hours for Stages I/II/III, against 900 for the RGB-only variant. Fig. 5 is measured on one A800 (bfloat16, batch size 1).
  • Scaling RGB instead. Extending RGB-only Stage II 3.7× to the full budget lifts randomized RoboTwin 2.0 from 0.81 to 0.83, and doubling demonstrations to 0.85, against 0.90 for World4Act (real robot: 0.76 vs. 0.85, Table 3(a)).

Table S5. Cost (A800 GPU-hours) and randomized RoboTwin 2.0 success.

VariantGPU-hoursSuccess ↑
RGB only900 (1.0×)0.81
RGB only, compute-matched2,520 (2.8×)0.83
RGB only, 2× demos1,200 (1.3×)0.85
World4Act2,520 (2.8×)0.90

Q3: What is the role of the 4D rollout, and why a generative 4D model rather than a lighter 3D objective?

Response to Q3: World4Act is one model with two roles, a 4D world simulator and a policy; the generative 4D formulation provides both.

  • Why generative. Video generation models carry strong priors that generalize across scenes. We add 4D understanding on top of this capability so that geometry and motion inherit its generalization; Tables 3(a) and S5 isolate the benefit.
  • Two roles. After Stage II, World4Act is already a 4D world simulator (Table 1) whose rollouts can augment data; once Stage III adds action conditioning (Eq. 7), they can also evaluate policies. As a policy it only outputs actions, so the rollout is skipped at deployment. A lighter 3D head offers no simulator, nor the Stage-II world-model pretraining whose removal costs 7 points (Table 3(b)).
  • Why 4D data. We believe that scaling 2D data cannot by itself bring 3D spatial understanding: in Table S5, more compute or demonstrations add 2–4 points, against 9 for 4D supervision. We built Robo4D_Dyn also to draw the community's attention to the importance of 4D data.

Thank you for conducting additional experiments and providing a cost breakdown. The evaluation against simulator "ground truth" largely addresses my previous concern—that the original 4D evaluation might merely have been measuring the consistency between the model and the annotated model. The two new real-world tasks also provide stronger evidence for the practical benefits of this approach in tasks involving spatial constraints. Furthermore, the comparison with a baseline model using only RGB data—under equivalent computational resource constraints—is highly informative for assessing the value of the additional training effort.

I still have two reservations. First, an ablation study using "only RGB data" (under identical training conditions) would more directly demonstrate that the performance gains are indeed attributable to the 4D supervisory signals. Second, the fact that removing "Stage II" does not immediately prove that the full generative 4D objective function is superior to a more lightweight geometric/motion prediction objective. Additionally, the mention of using "rollouts" for data augmentation and policy evaluation currently represents a potential application rather than a demonstrated practical benefit.

Furthermore, please clarify how the statement regarding "scaling up the RGB-only Stage II by 3.7x" relates to the "2,520 GPU-hour" computational budget mentioned in Table S5. While the results on compute-scaling do demonstrate the advantages of 4D supervision within the current evaluation setup, they are not sufficient to conclude that training based solely on RGB data has inherent, universal limitations regarding scalability.

Overall, this new evidence largely alleviates my initial concerns; therefore, I am adjusting my score from 2 to 3.

AuthorsOpenReview ↗

Response to Reviewer weNF

We thank the reviewer for the constructive suggestions and for raising the rating. Our responses to the remaining questions are below.

1. An RGB-only ablation under identical training conditions would show more directly that the gains come from the 4D signals.

Response: We ran this ablation as part of our experiments, including on the two new real-world tasks (Table S9). The RGB-only variant uses the same data, backbone and training schedule as World4Act and differs only in removing the 4D targets. Adding the 4D targets improves success by 9 points on randomized RoboTwin 2.0 and on the four original real-world tasks (Table 3(a)), and by 29 points on the two 3D-demanding tasks. These results show that training with 4D targets is indeed beneficial, most of all where 3D understanding is required.

Table S9. RGB-only ablation under identical training conditions; success (%). The first two columns are from Table 3(a); Vase and Cabinet are the two real-world 3D tasks of Table S4 (50 trials each).

VariantRoboTwin 2.0 (rand.)Real world (4 tasks)VaseCabinet3D avg.
RGB only8176544851
World4Act (RGB + 4D)9085827880

2. Removing Stage II does not show that the generative 4D objective is better than a lighter geometry/motion prediction objective.

Response: A main reason for building the 4D world model on a generative model is to inherit the strong generalization of video generation models. We are aware of lighter geometry-prediction models built on point transformers, such as PointWorld (Huang et al., 2026); when we tested this family, its predictions degraded on scenes and objects outside the training distribution far more than those of the video-generation backbone, which is why we did not build on it. The ablation without Stage II is meant to show the effectiveness of 4D supervision in world-model pretraining, not to compare objectives. The generative objective adds cost only in training: at deployment the rollout is skipped, and control latency is comparable to the RGB-based GigaWorld-Policy (Fig. 5). We will further explore lighter generative 4D models, building on the rapid progress in acceleration and distillation of generative models. We thank the reviewer for this suggestion.

3. Rollouts for data augmentation and policy evaluation are a potential application, not a demonstrated benefit.

Response: Using rollouts for data augmentation and policy evaluation is our ongoing work, and we are glad to share a first result (Table S10). On the two new real-world tasks, we train 3D Diffusion Policy (DP3) on 200 trajectories (100 per task) in two versions: with real point clouds, or with 4D data generated purely by our 4D world model. Both policies are evaluated on the real robot under the protocol of Table S4. The policy trained on generated 4D data achieves comparable results: 73% success, against 75% with real point clouds. This saves the time of real-robot data collection and confirms the effectiveness of the generated data; rollout-based policy evaluation is in progress.

Table S10. DP3 trained on real point clouds or on 4D data generated by World4Act (200 trajectories in total per source); real-world success (%), 50 trials per task.

DP3 training dataVaseCabinetAverage
Real point clouds787275
Generated 4D data (ours)747273

4. How does “scaling up the RGB-only Stage II by 3.7x” relate to the 2,520 GPU-hour budget in Table S5?

Response: The RGB-only variant costs 900 GPU-hours: 600 for Stage II and 300 for Stage III, with the same number of updates as World4Act (100k in Stage II); each update is cheaper because there are no 4D streams. The 4D pipeline costs 2,520: 1,000 for annotation and 1,520 for training. The compute-matched variant keeps Stage III unchanged and gives the remaining 2,220 GPU-hours to Stage II, i.e., 2,220/600=3.7×2{,}220 / 600 = 3.7\times (370k instead of 100k updates), for a total of 2,520. Thus 2.8×2.8\times (2,520/9002{,}520 / 900) is the ratio of the total budgets, and 3.7×3.7\times is the factor on Stage II alone.

5. The compute-scaling results do not show that RGB-only training has an inherent, universal scalability limit.

Response: Thank you for pointing this out. Our conclusion is specific to the current evaluation setup. We are also studying how RGB and 4D data compare as they scale, which we believe is an important line of research.

Reviewer_Grt5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. Incorporating Additional Modalities: Building upon the joint prediction of actions and future RGB frames, this approach adds the prediction of future point maps and 3D scene flows. By leveraging explicit 3D geometric and motion supervision, it enhances policy learning, while requiring only action generation during inference.
  2. Dataset Construction: Existing robot interaction data is integrated, with point maps and 3D scene flows automatically generated and temporally aligned. This results in the Robo4D Dyn dataset, comprising approximately 31.8 million frames, which serves to support 3D dynamic modeling and policy training.
Critical strengths
  1. Balancing training supervision with inference efficiency: Through an asymmetric attention design, the model leverages future multimodal supervision during training, while generating only actions during deployment, resulting in inference latency comparable to an RGB-only baseline.
  2. Comprehensive experimental evaluation: The approach is assessed across world prediction, simulated manipulation, and real world robotic tasks, with ablation studies analyzing the contributions of different prediction modalities, specialized VAEs, and staged training.
Critical weaknesses
  1. Insufficient analysis of the impact of biases in automated annotations: Since both training and evaluation rely on automatically generated 3D annotations, the model may inherit the systematic biases present in the annotation algorithms. Consequently, improvements in predictive metrics might, in part, reflect the model’s fitting to the outputs of these annotation algorithms rather than an accurate modeling of the actual 3D dynamics.
  2. Lack of sufficient statistical evidence for policy gains: Compared with GigaWorld-Policy, the proposed method improves the average real world success rate by only about 2 percentage points, without sufficient statistical evidence to establish the reliability of this gain.
Questions for authors

Given that the method proposed in this paper yields an improvement of only about 2 percentage points in average real world success rate compared to GigaWorld-Policy, it is recommended to further evaluate whether this gain exceeds the fluctuations attributable to randomness.

  1. GigaWorld-Policy: Report three success rates for each real-world task, one for each of three different random seeds, to assess the variability of the success rate.
  2. World4Act: Report three success rates for each real-world task, one for each of three different random seeds, to assess the variability of the success rate.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Grt5

We thank the reviewer for the constructive comments and address each point below.

W1: Training and evaluation both rely on automatically generated 3D annotations; the model may inherit their biases, and the gains may partly reflect fitting the annotation algorithm.

Response to W1: We would like to clarify that automatic annotation is the only way to obtain 4D supervision on real robot data, and that our gains hold against exact geometry that involves no annotation (Table S3).

  • Why an annotation model. No existing real-world manipulation dataset provides 4D information, i.e., dense 3D geometry and scene flow. To learn from real data, this supervision can only come from a 3D annotation model; we use 4RC with consistency checks (p. 12, L625–628).
  • Why not simulation. Simulators offer exact geometry, but their RGB differs markedly from real images, so we deliberately keep simulated data out of training. Robo4D_Dyn is itself a core contribution: we hope to provide the community with a large-scale real-robot dataset with 3D annotations.
  • Test against exact geometry. Simulation is, however, ideal for evaluation. We evaluate the Stage-II checkpoints of Table 1, without fine-tuning, on held-out clips of RoboTwin 2.0 and LIBERO, where point maps and scene flow come from the simulator. Neither benchmark is in our training data. World4Act lowers Relp\mathrm{Rel}_p by 12.9% / 10.0% and EPE3D\mathrm{EPE}_{3D} by 6.4% / 5.0% relative to RynnWorld-4D, and stays ahead of our projective variant trained on the same data. A gain obtained by fitting annotation-specific bias would not carry over to ground truth the model has never seen. The 4RC labels themselves are close to this ground truth (last row).
  • Annotation-free evidence. Policy success (Table 2) involves no 3D annotation.

Table S3. Stage-II prediction against simulator ground truth.

RoboTwin 2.0LIBERO
MethodRelp\mathrm{Rel}_p ↓EPE3D\mathrm{EPE}_{3D} ↓Relp\mathrm{Rel}_p ↓EPE3D\mathrm{EPE}_{3D} ↓
RynnWorld-4D0.3410.1560.3010.140
Projective 4D (same data)0.3120.1610.2870.151
World4Act0.2970.1460.2710.133
4RC annotations0.0410.0270.0350.022

W2 and Questions: The real-world gain over GigaWorld-Policy is only about 2 points; please report per-task success rates for three seeds for both methods.

Response to W2 and Questions: We report both methods on every real-world task for three training seeds (Table S1) and on RoboTwin 2.0 (Table S2).

  • Setup. Seeds 1–2 are the runs behind Table 2: their means reproduce its real-world entries, and their four-task averages, 82.5% and 84.5%, appear there rounded as 0.83 and 0.85. Seed 3 is new. Each seed re-runs policy training and is evaluated for 100 trials per task on the same initial configurations for both methods.
  • Real robot. The per-task standard deviation across seeds is 2.1–5.0 points, comparable to the binomial spread of 100 trials, so per-task differences are within seed variation. The four-task average is 82.42 ± 1.01% vs. 84.83 ± 1.61%: a gain of 2.4 ± 1.1 points across seeds (Welch p≈0.10p \approx 0.10), positive but not conclusive with three seeds. With the trained models fixed, a bootstrap over the 400 matched configurations gives a 95% interval of [+0.4, +4.4] points (p=0.02p = 0.02).
  • Simulation. With 2,500 episodes per seed and condition, World4Act gains 3.3 (clean) and 5.0 (randomized) points, more than four times the seed standard deviation, and every World4Act seed is above every GigaWorld-Policy seed.

Table S1. Real-world success (%) per task and seed (S1–S3); 100 trials per entry.

GigaWorld-PolicyWorld4Act
TaskS1S2S3Mean ± stdS1S2S3Mean ± stdΔ
Clean Desk87939190.33±3.0695899191.67±3.06+1.33
Scan QR73777675.33±2.0880768179.00±2.65+3.67
Sweep Trash78727474.67±3.0684748079.33±5.03+4.67
Stack Bowls88928889.33±2.3185939089.33±4.040.00
Average81.5083.5082.2582.42±1.0186.0083.0085.5084.83±1.61+2.42

Table S2. RoboTwin 2.0 success (%) over three seeds (S1–S3); 2,500 episodes per entry.

CleanRandomized
MethodS1S2S3Mean ± stdS1S2S3Mean ± std
GigaWorld-Policy87.8486.3686.7286.97±0.7784.5285.2485.6485.13±0.57
World4Act89.4490.8490.4890.25±0.7390.8889.2890.2090.12±0.80

Scope. We claim a gain in the task average, clear in simulation and moderate on the real robot, and none on individual tasks. We will report mean ± std over three seeds in Table 2, replace “83.0% to 85.0%” (p. 8, L410–412) with the three-seed averages, and describe the per-task gains as trends.

AuthorsOpenReview ↗

Response to Reviewer Grt5

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

AuthorsOpenReview ↗

Further discussion with Reviewer Grt5

Dear Reviewer Grt5,

Thank you for your thoughtful feedback on our paper. As the discussion period nears its end, we would like to see whether our previous responses have addressed your questions.

If any points would benefit from further clarification, please let us know. We would be glad to provide additional details.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

A brief summary of the reviews and rebuttal

We sincerely thank all reviewers and area chairs for their time and constructive feedback. Below, we briefly summarize the reviews and our responses.

All three reviewers (R1-Grt5, R2-weNF, and R3-NuDS) recognized the strengths of our work:

  1. R2 and R3 find the motivation sound and the representation well chosen: point maps and 3D scene flow are native, image-aligned prediction targets that reuse video-generation priors.
  2. All three reviewers highlight the action-centered causal design, which provides 4D supervision in training at no extra deployment cost.
  3. R1 and R3 commend the comprehensive evaluation and ablations, and all three reviewers find the paper clear and easy to follow.

In the rebuttal, we present additional experiments and clarifications:

  1. We evaluate 4D prediction against simulator ground truth on RoboTwin 2.0 and LIBERO, and explain why we annotate real data with 4RC.
  2. We report real-world and RoboTwin 2.0 results over three training seeds for GigaWorld-Policy and World4Act.
  3. We add two real-world tasks that require 3D perception, with comparisons to four RGB-only methods and an RGB-only ablation.
  4. We report annotation and training costs and compare with RGB-only training under a matched compute budget.
  5. We present action-intervention experiments on action–dynamics coupling, and isolate the contribution of 3D scene flow over 2D optical flow.
  6. We report the reconstruction accuracy of the adapted VAEs, clarify the training settings and the real-world protocol, and discuss the role of the 4D rollout.

We are grateful that R2 raised the rating from 2 to 3, noting that the new evidence largely alleviates the initial concerns. R1 found the simulator evaluation to offer strong evidence for the improved 4D prediction; the remaining concern on the accuracy of 4RC on real-world data is addressed in our follow-up response. We appreciate R3's detailed questions, though we did not receive a follow-up from R3 and so did not have a chance to discuss the new results further.

In summary, World4Act Policy brings 3D spatial awareness to World–Action Models by making point maps and 3D scene flow native prediction targets in training, while keeping low-latency, action-only inference at deployment.

We again thank all reviewers and area chairs for helping us improve our work.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.