acceptodds
Under review as a conference paper at ICLR 2027

Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising

Abstract

We propose X-WAM, a Unified 4D World Model that unifies real-time robotic action execution and high-fidelity 4D world synthesis (video + 3D reconstruction) in a single framework, addressing the critical limitations of prior unified world models (e.g., UWM) that only model 2D pixel-space and fail to balance action efficiency and world modeling quality. To leverage the strong visual priors of pretrained video diffusion models, X-WAM imagines the future world by predicting multi-view RGB-D videos, and obtains spatial information efficiently through a lightweight structural adaptation: replicating the final few blocks of the pretrained Diffusion Transformer into a dedicated depth prediction branch for the reconstruction of future spatial information. Moreover, we propose Asynchronous Noise Sampling (ANS) to jointly optimize generation quality and action decoding efficiency. ANS applies a specialized asynchronous denoising schedule during inference, which rapidly decodes actions with fewer steps to enable efficient real-time execution, while dedicating the full sequence of steps to generate high-fidelity video. Rather than entirely decoupling the timesteps during training, ANS samples from their joint distribution to align with the inference distribution. Pretrained on over 5,800 hours of robotic data, X-WAM achieves 79.2% and 90.7% average success rate on RoboCasa and RoboTwin 2.0 benchmarks, while producing high-fidelity 4D reconstruction and generation surpassing existing methods in both visual and geometric metrics.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, 4D World Modeling, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Jm6gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

X-WAM is a unified world-action model fine-tuned from a video DiT (Wan2.2-TI2V-5B). It consumes multi-view RGB, proprioceptive state, and language, and jointly predicts future RGB videos, depth videos, states, and actions. Two designs are central. First, a lightweight depth branch replicates the final M DiT blocks and reads the main branch through "unilateral attention" (depth attends to the main branch, and not vice versa); because the branch can be switched off at inference, action latency is unaffected. Second, Asynchronous Noise Sampling (ANS) couples the training noise levels of video and action within a joint distribution and adopts asynchronous denoising (actions in T_a steps, video in T_O), allowing actions to be dispatched early. The model is pretrained on roughly 5,874 hours of data and evaluated on RoboCasa, RoboTwin 2.0, 4D reconstruction/generation metrics, ablations, and real-robot tasks.

Critical strengths
  • A practical, latency-aware design. Decoupling a depth-supervised branch from the action path (so it can be disabled at inference) is a concrete answer to the "quality vs. real-time" tension, and the ablation in Table 3 backs the claimed latency/quality trade-off.
  • A coupled training/inference noise distribution. Framing the asynchronous schedule as a coupled sampling problem, and showing that decoupled-Async degrades reconstruction (Table 3b), is a clean and well-motivated contribution.
Critical weaknesses
  • The reconstruction comparison needs a more explicit common protocol. Table 2 reports strong RGB-D and point-cloud gains, but Section 4.2 does not make sufficiently clear whether methods are compared on the same action-conditioned future and how camera information is supplied. These details matter more to interpreting the reported gains than simply adding another benchmark; the current quantitative evidence is also confined to RoboCasa.
  • The control benefit under the full training recipe remains incompletely isolated. Table 3 supports a benefit from depth supervision without large-scale pretraining, whereas the real-robot comparison supports competitive performance rather than a causal improvement from depth modeling. The corresponding contribution after full pretraining is still not cleanly isolated.
Questions for authors
  • Reconstruction protocol. For Table 2, are all methods compared on the same action-conditioned future, or on each policy's own rollout? A matched-action comparison with shared calibration would isolate the geometry gains.
  • ANS: coupling vs. clean actions. Can the independent-sampling baseline in Table 3(b) receive the same clean-action probability p at fixed inference steps? This separates noise coupling from merely covering the late denoising regime.
  • Depth under full pretraining. Please add a depth/no-depth control under the full pretraining recipe, and clarify which shared parameters receive depth-loss gradients.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Jm6g

Thank you for the focused and constructive review. Short answers to the three questions come first; details follow. All new results are in the revised PDF.

  • Q1 (protocol). Each method's own rollout in the submission. In a new matched-action comparison with shared calibration, the geometry gap is unchanged: AbsRel −67%, CD −88% (Table 2(b)).
  • Q2 (coupling vs. clean actions). With the same clean-action probability p and the same inference steps, coupling still improves all six generation metrics, so it does more than cover the late denoising regime (Table 3(c)).
  • Q3 (depth under full pretraining). Removing only the depth branch from the full recipe lowers RoboCasa success from 79.2 to 74.1 (Sec. 4.1).

W1 / Q1. A common protocol for the reconstruction comparison.

Same future or own rollout. Each method was evaluated on its own closed-loop rollout: it executed its own predicted actions, and its predictions were compared with the frames the simulator rendered along that rollout. Robot4DGen does not provide code for its action pipeline, so we ran it through an external inverse dynamics module that predicts actions from its generated frames. We omitted this from the submission and apologize; Sec. 4.2 now describes the protocol.

Matched actions, shared calibration. The baselines cannot be conditioned on given actions, so the comparison the reviewer suggests is possible between X-WAM and X-WAM w/o depth + DA3. On 2,400 held-out clips, both start from the same frame, receive the same ground-truth actions, and use the same ground-truth calibration, including the wrist pose:

Matched actions, shared calibrationPSNR↑SSIM↑LPIPS↓AbsRel↓δ1↑CD↓
X-WAM w/o depth + DA323.410.90150.04710.09620.91740.0358
X-WAM23.920.90240.04640.03180.97620.0044

AbsRel falls by 67% and CD by 88%, the same relative gaps as under the original protocol (0.1045 → 0.0349 and 0.0401 → 0.0049). The geometry gain therefore comes neither from different trajectories nor from calibration. What remains is the depth branch itself, trained with in-domain depth, against a zero-shot estimator; architectures under equal depth supervision are compared in Table 3(a).

Camera information (now in Sec. 4.2).

  • X-WAM and X-WAM w/o depth + DA3 use ground-truth intrinsics and static-camera extrinsics, and place the wrist camera with their own predicted end-effector pose and the hand-eye calibration. In the latter, DA3 only estimates depth.
  • DreamZero + DA3 uses cameras estimated by DA3, so its Chamfer distance also reflects camera-estimation error. The revised text no longer attributes its whole gap to joint modeling.
  • Robot4DGen predicts XYZ maps, which are used directly. It also takes the current RGB-D frame as input.

W2 / Q3. Depth under the full training recipe; gradient path.

Alongside the full X-WAM, we had also pretrained a variant without the depth branch, with the same 5,874-hour corpus, schedule and hyperparameters. Fine-tuned on RoboCasa as in Table 1:

RoboCasa SR (2,400 episodes)w/o depthwith depthGain
No robotic pretraining (Table 3(a))63.067.8+4.8
Full pretraining (Table 1 setting)74.179.2+5.1

The depth gain has the same size with and without pretraining.

Which parameters receive depth gradients. The depth loss is back-propagated without stop-gradient. It updates the M depth blocks and the depth head and, through the cross-attention of Eq. (3), the shared trunk (blocks 1 to N−M) and main-branch blocks N−M+1 to N−1. The last main block and the action, state and video heads receive no depth gradient. The main branch's forward pass never reads depth tokens, so the branch can be switched off at inference.

Q2. ANS: coupling vs. clean actions.

We crossed noise coupling with p at fixed inference steps (Ta=5T_a = 5, TO=25T_O = 25; each cell is SR / PSNR / AbsRel / CD):

Training noisep = 0p = 0.5
Independent67.2 / 22.60 / 0.0430 / 0.006167.7 / 23.01 / 0.0371 / 0.0052
Coupled (Eq. 4)67.3 / 22.95 / 0.0402 / 0.005767.8 / 23.46 / 0.0349 / 0.0049 (ANS)

With p matched, coupling improves all six reconstruction metrics at either value (at p = 0 / 0.5: PSNR +0.35 / +0.45 dB, AbsRel −6.5% / −5.9%, CD −6.6% / −5.8%). Clean-action exposure contributes as well (AbsRel and CD about −14% under either scheme), and the two effects are additive. Both are parts of Eq. (4), its second and first case, and correspond to the two phases of asynchronous inference. The four success rates are close (67.2–67.8): the benefit of ANS is generation quality, at the 4.5× lower action latency of asynchronous inference.

In sum, the geometry gain is now isolated from trajectories and calibration (Table 2(b)), and the control gain from pretraining (Sec. 4.1). We are glad to clarify anything further.

Thank you for the detailed rebuttal and additional experiments. The full-experiments address my concerns. I will therefore keep my positive score.

We sincerely thank the reviewer for their thoughtful feedback and positive assessment of our work. We greatly appreciate your time and constructive suggestions throughout the discussion.

Reviewer_sdXJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces X-WAM, a world-action model initialized from Wan2.2 that jointly predicts future multi-view RGB videos, robot states, and actions, with an auxiliary depth branch supporting subsequent 3D reconstruction. Its main contributions are an interleaved depth branch constructed by duplicating the final Transformer blocks and Asynchronous Noise Sampling (ANS), which couples training noise levels to enable earlier action decoding while video denoising continues. The model is pretrained on approximately 5,874 hours of robotic data and evaluated on RoboCasa, RoboTwin 2.0, RGB-D reconstruction metrics, and real-robot manipulation tasks. The reported simulation success rates and reconstruction performance are promising.

Critical strengths
  1. The depth branch can be disabled during policy execution, allowing geometric supervision to benefit shared representations without requiring depth generation at deployment. Table 3(a) shows an increase in success rate from 63.0% to 67.8% with unchanged action latency. This provides meaningful evidence that auxiliary geometry can improve control within the proposed architecture.

  2. The paper evaluates policy performance on two substantial manipulation benchmarks and reports visual, depth, and point-cloud reconstruction metrics. Evaluating both capabilities is particularly relevant to a model intended to unify world modeling and action generation. The strong reported simulation results warrant further consideration of the approach.

Critical weaknesses
  1. TesserAct [1] adapts a pretrained video model to generate RGB, depth, and normals, reconstructs 4D scenes, and uses these predictions for action inference. Robot4DGen [2] jointly predicts appearance and geometry with cross-view consistency and derives executable trajectories from generated RGB-D videos. MVISTA-4D [3] combines view-consistent RGB-D generation with test-time action inference using a Wan2.2 backbone. Together, these methods provide substantial precedent for using geometric future prediction to support manipulation.

    X-WAM’s direct integration of action generation and optional depth prediction is a useful architectural distinction. However, its contribution appears to lie primarily in this integration and the proposed noise-sampling heuristic. Independent modality-specific noise levels and flexible conditional generation are already explored in UWM [4] and Motus [5]. The methodological advance therefore appears narrower than the paper’s framing as a new unified 4D modeling paradigm suggests. This makes controlled evidence of an advantage over closely related designs particularly important.

  2. Table 2 includes Robot4DGen [2], but omits TesserAct [1] and MVISTA-4D [3], both of which directly connect geometric future prediction with downstream control. The post-hoc DA3 pipelines offer useful reference points, but the manuscript does not clarify whether their depth estimators receive comparable task-specific depth supervision. Their reconstruction gap therefore does not cleanly isolate the benefit of the interleaved architecture. Furthermore, all controlled ablations exclude large-scale robotic pretraining, leaving unclear whether the observed component-level benefits explain the full model’s headline improvements.

    The ANS experiments also do not sufficiently distinguish its contribution from simpler alternatives. In Table 3(b), independent-noise training with asynchronous inference already achieves the same 1,033 ms latency. ANS increases success from 67.2% to 67.8%, but no uncertainty estimates are reported. It also assigns a 50% probability to generation conditioned on clean actions, without a control separating this choice from the proposed noise coupling. The generation-quality gains could therefore partly arise from explicit training exposure to clean actions. Finally, Eq. (4) imposes an ordering on noise levels, but this alone does not establish the claimed exact match to the distribution induced by the inference schedule. These gaps leave both the necessity and the distinct contribution of ANS unresolved.

[1] Haoyu Zhen et al. TesserAct: Learning 4D Embodied World Models. arXiv:2504.20995, 2025.

[2] Zeyi Liu et al. Geometry-aware 4D Video Generation for Robot Manipulation. arXiv:2507.01099, 2025.

[3] Jiaxu Wang et al. MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation. arXiv:2602.09878, 2026.

[4] Chuning Zhu et al. Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets. arXiv:2504.02792, 2025.

[5] Hongzhe Bi et al. Motus: A Unified Latent Action World Model. arXiv:2512.13030, 2025.

Questions for authors
  1. Which baselines in Tables 1 and 2 were retrained, and what backbones, robotic pretraining data, downstream data, and depth supervision did they use? In particular, were the DA3 pipelines adapted using the same depth annotations available to X-WAM?

  2. Can you compare the interleaved branch with a simpler auxiliary depth head under matched training budgets, including robotic pretraining? This would help determine whether the gains depend on the proposed architectural design. Please also explain why TesserAct [1] and MVISTA-4D [3] were omitted from the geometric prediction comparisons.

  3. Does the baseline with independently sampled noise use the same 50% probability of clean-action-conditioned training? If not, can you provide this control, together with uncertainty estimates for policy performance? Please also clarify whether “matching the inference distribution” refers to matching its support or its full probability distribution. If the latter, how does Eq. (4) establish this match?

[1] Haoyu Zhen et al. TesserAct: Learning 4D Embodied World Models. arXiv:2504.20995, 2025.

[2] Zeyi Liu et al. Geometry-aware 4D Video Generation for Robot Manipulation. arXiv:2507.01099, 2025.

[3] Jiaxu Wang et al. MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation. arXiv:2602.09878, 2026.

[4] Chuning Zhu et al. Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets. arXiv:2504.02792, 2025.

[5] Hongzhe Bi et al. Motus: A Unified Latent Action World Model. arXiv:2512.13030, 2025.

5 comments
AuthorsOpenReview ↗

Response to Reviewer sdXJ (1/2)

Thank you for the careful review. The review locates our contribution in the integration of depth prediction with joint video–action generation and in the noise-sampling design, and notes that this "makes controlled evidence of an advantage over closely related designs particularly important." We agree, and we ran the three controls the review asks for. All are in the revised PDF.

Requested controlResultRevised paper
Simpler auxiliary depth head (Q2)63.9% success, vs. 67.8% for the interleaved branch and 63.0% without depthTable 3(a)
Component benefit under robotic pretraining (W2)Depth adds 5.1 points under full pretraining (74.1 → 79.2) and 4.8 withoutSec. 4.1
Independent noise with the same clean-action probability (Q3)With p matched, coupling still improves all six generation metricsTable 3(c)

W1. Relation to TesserAct, Robot4DGen, MVISTA-4D, UWM and Motus.

Our claim (Sec. 1 and 2.2) is specific: X-WAM brings explicit geometry into unified world action models, which denoise video and actions jointly. The table shows what this adds to each line of work (times: first three rows as reported in each paper, on its own hardware; last two from Table 6).

Predicts geometryHow actions are obtainedTime to an action
TesserActyespoint-cloud inverse dynamics model, after generationabout 16 s
Robot4DGenyesgripper tracking on the generated videoabout 30 s
MVISTA-4Dyestest-time optimization through the frozen generator47–98 s
UWM, Motusnojoint denoising with video380 ms (Motus)
X-WAMyesjoint denoising with video304 ms

The 4D world models generate geometry first and extract actions afterwards. Unified WAMs act in real time, but in 2D. X-WAM combines the two: depth shapes the shared representation as training supervision and is switched off during control, so control gains about 5 points (W2.2) at unchanged latency. The revised Sec. 2.2 states this distinction. UWM and Motus sample each modality's noise level independently (our decoupled baseline); W2.3 separates what ANS changes relative to it.

W2.1 / Q2. TesserAct and MVISTA-4D; DA3 supervision; simpler depth head.

TesserAct and MVISTA-4D. Neither is designed for our camera setting. TesserAct generates a single view, a limitation its authors acknowledge, whereas Table 2 evaluates a point cloud fused from several views, including a moving wrist camera. MVISTA-4D is designed for static cameras: it takes the extrinsics of every view as input and samples its cross-view attention along epipolar lines computed from them, and the future poses of our wrist camera are unknown at inference. We therefore compare with their core design choice on a matched backbone and data. TesserAct sums per-modality projections of the RGB and depth latents into the same tokens, which equals channel concatenation followed by a linear embedding. The channel-concatenation variant of Table 3(a) is thus a TesserAct-style fusion (without normals) on our backbone.

DA3. DA3 is used zero-shot, without the depth annotations X-WAM is trained on. The DA3 rows of Table 2 therefore compare X-WAM with a general-purpose post-hoc estimator. The architecture is isolated in Table 3(a), where every variant is trained with the same ground-truth depth.

Simpler depth head. The new row is an MLP that predicts depth from the last-layer features of the main branch, trained with the same loss and without stop-gradient. No robotic pretraining; 2,400 episodes per success rate (SR):

Depth designSRAction latency (ms)AbsRel↓CD↓
No depth63.01033––
Simple depth head (new)63.910330.04550.0072
Channel concat. (TesserAct-style)64.212660.03770.0052
Interleaved branch (ours)67.810330.03490.0049

Depth supervision alone does not improve control: the simple head stays at the no-depth level (+0.9) and gives the worst geometry. The interleaved branch is 3.9 and 3.6 points above the simple head and TesserAct-style fusion (z = 2.9 and 2.6), with better geometry and no added latency. What matters is how depth enters the network. All variants share data, schedule and initialization. Repeating the 5,874-hour pretraining for every variant is beyond what we can run during the rebuttal; under full pretraining we have the depth / no-depth pair in W2.2.

W2.2. Controlled ablations under robotic pretraining.

Alongside the full X-WAM, we had also pretrained a variant without the depth branch, with the same corpus, schedule and hyperparameters. After the same RoboCasa fine-tuning it reaches 74.1%, against 79.2% with depth (p < 0.001). The gain is 5.1 points under full pretraining and 4.8 without. The component benefit therefore carries over to the full model, and 5,874 hours of pretraining do not wash it out.

AuthorsOpenReview ↗

Response to Reviewer sdXJ (2/2)

W2.3 / Q3. What ANS contributes beyond simpler alternatives.

What ANS is for. We agree that the 1,033 ms latency comes from asynchronous inference and is shared by the decoupled baseline. Asynchronous inference has a cost in generation quality, however, and this is what ANS addresses (Table 3(b)):

Training → inferenceAction latency (ms)PSNR↑AbsRel↓CD↓
Synchronous → synchronous466523.480.03750.0041
Independent → asynchronous103322.600.04300.0061
ANS → asynchronous103323.460.03490.0049

With independent noise, the asynchronous model is 0.88 dB below the synchronous one in PSNR, and its AbsRel is 15% higher. ANS recovers the PSNR (23.46 vs. 23.48), gives the lowest depth error and closes most of the Chamfer gap, at the same 4.5× lower action latency.

Clean-action control. The independent baseline used p = 0. We completed the 2×2 the review asks for (asynchronous inference; each cell is SR / PSNR / AbsRel / CD):

Training noisep = 0p = 0.5
Independent67.2 / 22.60 / 0.0430 / 0.006167.7 / 23.01 / 0.0371 / 0.0052
Coupled (Eq. 4)67.3 / 22.95 / 0.0402 / 0.005767.8 / 23.46 / 0.0349 / 0.0049 (ANS)

As the reviewer anticipated, exposure to clean actions explains part of the gain (AbsRel and CD about −14% under either sampling scheme). Coupling contributes separately: with p matched, it improves all six reconstruction metrics (at p = 0 / 0.5: PSNR +0.35 / +0.45 dB, AbsRel −6.5% / −5.9%, CD −6.6% / −5.8%). The two effects are additive, and both are parts of Eq. (4): its first case is the clean-action mode and its second case is the coupling. They correspond to the two phases of asynchronous inference: joint denoising with tO≥tat_O \ge t_a up to step TaT_a, then video-only denoising with clean actions.

Uncertainty. With 2,400 episodes per entry, the 95% CI of a success rate is about ±1.9 points. All asynchronous variants lie between 66.9% and 67.8%, so we no longer claim a success-rate advantage for ANS (revised Sec. 4.3): its benefit is generation quality at unchanged latency. The depth gains are outside this margin (63.0 → 67.8, z = 3.5; 74.1 → 79.2, z = 4.2).

Support or distribution. Support. The wording in Sec. 1, "strictly matches the test-time distribution", was too strong; the revised abstract and Sec. 1 state the claim in terms of support. The inference schedule is linear in the step index: at joint step k≤Tak \le T_a, tO=1−k/TO≥1−k/Ta=tat_O = 1 - k/T_O \ge 1 - k/T_a = t_a, and afterwards ta=0t_a = 0. Every inference state therefore lies in the support of Eq. (4), for any ratio Ta/TOT_a/T_O, which is why one model serves different schedules. Independent uniform sampling puts half of its mass on tO<tat_O < t_a, which inference never visits, and none on ta=0t_a = 0.

Q1. Baseline training and depth supervision (now in Appendix B.3).

  • Table 1. π0 and π0.5 are taken from published results. Spatial Forcing is reproduced with its official code. GigaWorld-Policy and Motus are run on RoboCasa with their official code. These baselines use their own large-scale robotic pretraining. UWM is reimplemented on our Wan2.2-TI2V-5B backbone, without robotic pretraining.
  • Table 2. No method uses robotic pretraining, X-WAM included (its rows are the Table 3 models). DreamZero is trained from the same Wan2.2-TI2V-5B initialization on the same RoboCasa data. Robot4DGen keeps its original setting, with the current RGB-D frame as input, i.e., ground-truth depth that X-WAM does not receive.
  • Depth supervision. X-WAM is trained with simulator depth. DA3 is used zero-shot and was not adapted with these annotations.

The controls sharpen what the paper shows. (i) Whether depth supervision helps control depends on where it is attached: +0.9 points for an auxiliary head, +1.2 for input-level fusion, +4.8 for the interleaved branch, the last at no latency cost. (ii) The gain persists under 5,874 hours of pretraining (+5.1). (iii) Asynchronous decoding degrades generation under independent noise, and each component of ANS recovers part of it. The claims on ANS are narrowed accordingly. If a concern remains that would still weigh against acceptance, we would be glad to address it during the discussion.

Dear reviewer sdXJ,

As the discussion period comes to an end, we would be grateful to hear whether our responses have addressed your main concerns. If there are any remaining questions or points that would benefit from further clarification, please let us know. We would be happy to provide additional information.

Reviewer_7NvZOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes X-WAM, a unified 4D World Action Model that jointly predicts future multi-view RGB-D videos, robot proprioceptive states, and actions from a pretrained video diffusion model. To incorporate 3D spatial structure without doubling the token sequence length, the authors replicate the final few blocks of the Diffusion Transformer into an interleaved depth branch connected via unilateral cross-attention. To reduce action inference latency, they introduce Asynchronous Noise Sampling (ANS), which decodes actions in fewer denoising steps than video at test time and couples the video and action noise levels during training. Experiments on simulation benchmarks (RoboCasa and RoboTwin 2.0) and real-world bimanual manipulation tasks show that the method achieves competitive control performance and accurate 4D spatial reconstruction.

Critical strengths
  1. Practical and well-motivated framework design: Replicating the final Diffusion Transformer blocks with unilateral cross-attention provides a clean way to add depth supervision without altering the pretrained RGB trunk or increasing token sequence length, and the depth branch can be turned off during online control to avoid extra latency.
  2. Comprehensive empirical evaluation: The paper evaluates the model across diverse dimensions, including closed-loop policy execution on RoboCasa and RoboTwin 2.0, multi-view 4D reconstruction metrics, matched-backbone latency analysis, and real-world bimanual manipulation under visual domain shifts.
Critical weaknesses
  1. Architectural assumptions in unilateral depth adaptation and asynchronous denoising:

    • Role of the depth branch at inference and noisy depth supervision: Because the depth branch only branches off at the final M = 10 blocks with unilateral attention (the main branch never reads from the depth branch in Equation 3) and is toggled off during online policy execution, the model does not explicitly use reconstructed 3D geometry during control. Instead, spatial awareness relies entirely on auxiliary gradient supervision into the shared trunk during training. In addition, during the 5,874-hour pretraining stage, depth targets are pseudo-labels extracted by a monocular estimator (Video Depth Anything), which may lack multi-view metric consistency.
    • Video-to-action guidance and distribution matching in ANS: Under the asynchronous inference schedule (e.g., T_a = 10 action steps and T_O = 50 video steps), actions are completely decoded while the video latents only progress from t_O = 1.0 to 0.8 (still containing 80%–100% noise). From my understanding, it seems that future visual predictions are still very blurry when action decoding finishes, which raises the question of how much fine-grained future imagination actually guides the actions at test time. Furthermore, while the paper states that ANS strictly matches the test-time distribution, the inference schedule follows a deterministic 1D line segment in [1 - T_a/T_O, 1] followed by t_a = 0, whereas Equation 4 samples the entire upper triangle with a Beta(1.5, 1) prior and allocates p = 0.5 of training steps to clean actions (t_a = 0, where the action branch receives no denoising supervision even though online policy rollouts terminate at step T_a).
  2. Experimental controls, baseline fairness, and anomalies in reported results:

    • Pretraining scale vs. architectural gains (Table 1 vs. Table 3): In Table 1, X-WAM achieves 79.2% average success rate on RoboCasa, outperforming GigaWorld-Policy (68.9%) and Motus (67.6%) by over 10 points. However, in Table 3 (where large-scale pretraining is omitted), the full X-WAM model achieves 67.8%, which is on par with Spatial Forcing (67.8%) and Motus (67.6%) and slightly below GigaWorld-Policy (68.9%). This suggests that the large margin in Table 1 is primarily driven by the 5,874-hour pretraining corpus (which also includes RoboCasa MimicGen and RoboTwin 2.0 trajectories in Table 4) rather than the architectural modifications alone, whereas the baselines in Table 1 do not share this pretraining corpus.
    • Identical metrics between Table 2 and Table 3 and evaluation protocol in Section 4.2: All six reconstruction metrics for "X-WAM (Ours)" in Table 2 (23.46 PSNR, 0.8942 SSIM, 0.0513 LPIPS, 0.0349 AbsRel, 0.9738 delta_1, 0.0049 CD) are identical to four decimal places to those of the un-pretrained ablation model in Table 3 (which uses T_a = 5, T_O = 25 instead of T_a = 10, T_O = 50). Similarly, "X-WAM w/o depth + DA3" in Table 2 has the exact same RGB metrics as "No depth" in Table 3(a). It is also unclear how the baselines in Table 2 (such as Robot4DGen, which is a video generator without a policy) are evaluated when Section 4.2 states that metrics are computed by executing the policy in simulation. Moreover, pixel-level depth metrics exclude the dynamic wrist camera due to end-effector pose errors, and no standalone accuracy metric for the predicted proprioceptive states s_{1:H} (Equation 2) is reported.
    • Unisolated real-robot comparisons (Appendix C): While the authors candidly note in Appendix C.2 that the progress gap over pi_0.5 (70.7% vs. 69.2% in Table 5) is small relative to the 20-trial budget and that attributing OOD robustness to depth is observational because the models differ in multiple factors, leaving "X-WAM w/o depth" untested on the physical robot weakens the real-world validation of 4D modeling. In addition, the flagship Earphone Packing task (Appendix C.1, trained on 20 hours of data to showcase sub-centimeter 3D reasoning) only presents qualitative rollouts without quantitative success rates or baseline comparisons.
  3. Writing, formatting, and presentation:

    • Ambiguity in the depth branch formulation: Section 3.2 (lines 262–265) states that depth maps are replicated to three channels, encoded into the causal VAE latent space, denoised, and decoded via the VAE decoder. However, Equation 1 contains no noisy depth latent in the input sequence, and Algorithm 1 (Line 13) and Equation 6 formulate the depth output as direct one-step MSE regression (\hat{D} <- Head_depth(Z_D)) from noisy hidden states at arbitrary video noise levels t_O. Clarifying whether the depth branch predicts velocity in VAE latent space or directly regresses clean inverse depth in latent/pixel space would make Section 3.2 much clearer.
    • Placement of core empirical evidence and minor text issues: Although real-time physical deployment is highlighted as a core contribution in Figure 1 and Section 1, all real-robot experiments (Appendix C) and end-to-end latency comparisons against other WAMs (Appendix D) are deferred to the appendix. Also, the caption of Table 1 mentions that "blank cells indicate that the method has not been evaluated," although no cells in Table 1 are blank.
Questions for authors
  1. Regarding the depth branch formulation in Section 3.2 and Algorithm 1: Since there is no noisy depth token in the input sequence Z (Equation 1), does Head_depth(Z_D) directly regress the clean VAE latent of inverse depth in a single step regardless of the current video noise level t_O? At test time during 4D generation, is the depth branch only executed at the final denoising step T_O?
  2. Under Asynchronous Noise Sampling, actions are fully decoded during the first T_a steps while the video latents remain at high noise levels (t_O in [0.8, 1.0] for T_a = 10, T_O = 50). How much predictive geometric or visual signal can the action tokens extract from such noisy video latents? Have the authors tested varying the action-conditioned sampling probability p in Equation 4 (especially since online policy rollouts stop at step T_a and never run the t_a = 0 phase)?
  3. Comparing Table 1 (79.2% on RoboCasa with 5,874h pretraining) and Table 3 (67.8% without pretraining), most of the performance gain over prior WAMs (e.g., GigaWorld-Policy at 68.9% and Motus at 67.6%) appears to come from the large-scale pretraining stage. Could the authors clarify how the baselines in Table 1 were trained, and what the RoboCasa 4D reconstruction metrics in Table 2 look like for the fully pretrained X-WAM checkpoint (since Table 2 currently reports the exact same numbers as the un-pretrained checkpoint in Table 3)?
  4. In Section 4.2 (Table 2), how are non-policy baselines like Robot4DGen evaluated if X-WAM is evaluated during closed-loop policy execution where different trajectories lead to different ground-truth states? In addition, could the authors report the pose error of the predicted end-effector state s_{1:H} used in Equation 2, as well as quantitative results (or an ablation without the depth branch) on the real-robot tasks in Appendix C?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7NvZ (1/2)

Thank you for the detailed and careful review. Short answers to the four questions come first; the evidence follows under the corresponding weaknesses. All new results are in the revised PDF.

  • Q1 (depth formulation). Yes to both. The depth head regresses the clean VAE latent of inverse depth in one step, at any tOt_O, and there is no noisy depth token. In 4D generation the branch runs only at the final step (W3.1).
  • Q2 (ANS). The action tokens read a coarse future from the video hidden states, and this is sufficient: the same checkpoint scores 66.3% when actions finish with a fully denoised video and 67.2% when they finish at tO=0.8t_O = 0.8. Varying p over {0, 0.25, 0.5, 0.75} keeps success within 66.9–67.8 (W1.2).
  • Q3 (pretraining). New control: under identical pretraining, removing only the depth branch gives 74.1 vs. 79.2, the same gain as without pretraining (+5.1 vs. +4.8). The fully pretrained checkpoint improves all six Table 2 metrics (W2.1, W2.2).
  • Q4 (protocol, state error, real robot). Every method was evaluated on its own closed-loop rollout; a new matched-action comparison leaves the geometry gap unchanged. The predicted end-effector state has a mean error of 1.38 cm and 2.73°.

W1.1. Role of the depth branch at inference; pseudo-depth labels.

The reviewer's description is accurate: depth acts through training supervision, and control reads no reconstructed geometry. This is deliberate. Control needs only multi-view RGB, proprioception and language, with no depth sensor, calibration or reconstruction at test time, and action latency is unchanged (1,033 ms with or without the branch). The supervision is still effective: the depth loss is back-propagated without stop-gradient, through the cross-attention of Eq. (3), into the shared trunk (now stated in Sec. 3.2), and it improves control by about 5 points without and with robotic pretraining. For the second row we had also pretrained a variant without the depth branch, with the same corpus, schedule and hyperparameters as the full model (new, Sec. 4.1):

RoboCasa SR (2,400 episodes)w/o depthwith depthDepth gain
No robotic pretraining (Table 3(a))63.067.8+4.8
Full pretraining (Table 1 setting)74.179.2+5.1

We agree that the pseudo-labels lack multi-view metric consistency; the Limitations now state this. They are relative inverse depth, normalized per video, and serve only as a representation prior during pretraining. The models of Tables 2 and 3 are trained without pretraining, on simulator depth, so those results involve no pseudo-labels.

W1.2 / Q2. Video-to-action guidance; distribution matching; clean actions.

Guidance. The reviewer's calculation is right: actions finish at tO=1−Ta/TO=0.8t_O = 1 - T_a/T_O = 0.8. What the action tokens read is the clean current observation and the hidden states of the video tokens, which at this noise level carry a coarse prediction of the future, not a rendered video. This is the premise of asynchronous decoding (Sec. 1: actions can be recovered from highly noisy video latents), and Table 3(b) confirms it for X-WAM: the same decoupled-trained checkpoint reaches 66.3% with synchronous inference, where actions finish together with a fully denoised video, and 67.2% with asynchronous inference. A cleaner video at decision time does not improve control; the remaining steps serve 4D generation.

Distribution matching. "Strictly matches the test-time distribution" was inaccurate and is corrected (abstract and Sec. 1): what Eq. (4) matches is the support. Before step TaT_a, inference follows the segment the reviewer describes, which lies in {tO≥ta}\lbrace t_O \ge t_a \rbrace; independent sampling spends half of its samples on tO<tat_O < t_a, which never occurs. After step TaT_a, actions are clean while the video is still noisy; independent sampling never produces this state, and the first case of Eq. (4) trains it. Eq. (4) covers the whole region rather than one segment so that one model serves different schedules.

Clean actions (p = 0.5). Rollouts indeed stop at step TaT_a. The ta=0t_a = 0 samples train the video-only phase that follows when a complete video is wanted, and they carry no action or state loss (now stated in Sec. 3.4 and Algorithm 2). The sweep the reviewer asks for is in Table 3(c):

p00.250.5 (paper)0.75
SR67.367.667.866.9
AbsRel↓0.04020.03560.03490.0347

Reconstruction improves mainly from p = 0 to 0.25 and is flat afterwards. Success is unaffected even at p = 0.75, where only a quarter of the samples supervise the action branch.

AuthorsOpenReview ↗

Response to Reviewer 7NvZ (2/2)

W2.1 / Q3. Pretraining scale vs. architectural gains.

Pretraining and the depth branch both matter, and the new control separates them (table in W1.1): within each row, where everything else is identical, the depth branch adds about 5 points. On the comparison with the baselines:

  • How the Table 1 baselines were trained (Appendix B.3). π0 and π0.5 are taken from published results; Spatial Forcing, GigaWorld-Policy and Motus are run with their official code. All of these use their own large-scale robotic pretraining; only our UWM reimplementation does not. Table 1 thus compares pretrained models with pretrained models, and X-WAM without any robotic pretraining (67.8) is already on par with them (68.9 and 67.6).
  • Pretraining adds no benchmark data. The RoboCasa MimicGen and RoboTwin 2.0 parts of the corpus are exactly the fine-tuning sets that the baselines also train on (Appendix B.1).
  • At matched backbone and data, X-WAM without pretraining exceeds our Wan2.2-TI2V-5B reimplementation of UWM by 7.0 points (67.8 vs. 60.8).

W2.2 / Q3, Q4. Identical metrics in Tables 2 and 3; evaluation protocol.

Identical numbers. They are the same models, intentionally, and we failed to say so. The Table 2 baselines have no robotic pretraining, so Table 2 uses X-WAM trained without it, at Ta=5T_a = 5, TO=25T_O = 25: "X-WAM" is the interleaved-branch model of Table 3, and "X-WAM w/o depth + DA3" is its "No depth" model with DA3 applied to the RGB output. Sec. 4.2 now states this. As requested, under the same protocol and inference setting the fully pretrained checkpoint reaches PSNR 24.20, SSIM 0.9058, LPIPS 0.0424, AbsRel 0.0308, δ1 0.9872 and CD 0.0041, better than the Table 2 row on all six metrics (added to Table 2 as a reference row).

How the baselines were evaluated. Every method was evaluated along its own closed-loop rollout: it executed its own predicted actions, and its predictions were compared with the frames the simulator rendered along that rollout. Robot4DGen does not provide code for its action pipeline, so we executed it through an external inverse dynamics module, trained on the demonstrations, that predicts actions from its generated RGB frames. The generator stays frozen, and the metrics evaluate its own generated RGB-D. We omitted this from the submission and apologize; it is now in Sec. 4.2 and Appendix B.3.

Do different trajectories explain the gap? The baselines cannot be conditioned on given actions, so we tested this for the two models that can, X-WAM and X-WAM w/o depth + DA3. On 2,400 held-out clips, both receive the same ground-truth actions and the same ground-truth calibration, including the wrist pose (new Table 2(b)). The depth branch reduces AbsRel from 0.0962 to 0.0318 (−67%) and CD from 0.0358 to 0.0044 (−88%), the same relative gaps as in the closed-loop protocol.

Wrist camera and state accuracy. The predicted end-effector states s1:Hs_{1:H} have a mean error of 1.38 cm and 2.73° over the 8 predicted steps, reaching 2.2 cm and 4.6° at the last step (new Appendix E). A rotation error of a few degrees shifts the wrist image by several pixels, which is why the pixel-level metrics use the static cameras only. Chamfer distance covers the wrist camera and includes this pose error.

W2.3 / Q4. Real-robot comparisons.

We agree that the real-robot evidence for depth is observational, as Appendix C.2 notes and the Limitations now state. A real-robot ablation without the depth branch is under way. The original experiments were run some time ago, so we are first restoring a setup in which the two models can be compared fairly. We will post the result here if it is ready within the discussion period, and will add it to the paper in any case. Until then, the controlled evidence for depth under the full training recipe is the RoboCasa result (79.2 vs. 74.1). Earphone packing is a qualitative demonstration, and Appendix C.1 now presents it as such; the quantitative real-robot results are those of Table 5.

W3.1 / Q1. Depth branch formulation.

There is no noisy depth latent, consistent with Eq. (1). In a single step, the depth head regresses the clean VAE latent of inverse depth for all frames (observed and future), at whatever video noise level tOt_O the trunk receives. It predicts neither a velocity nor pixels; the VAE decoder maps the latent to inverse-depth maps. In Sec. 3.2, "after denoising" referred to the video denoising process: in 4D generation the branch runs once, at the final step, and during policy execution it never runs. Sec. 3.2 and Appendix B.2 are rewritten accordingly.

W3.2. Placement of evidence; Table 1 caption.

The caption is corrected. The real-robot experiments and the latency comparison remain in Appendices C and D: the additional page of the revision went to the controls requested above.

We hope these controls resolve the soundness concerns, and we are glad to clarify anything further.

Dear reviewer 7NvZ,

As the discussion period comes to an end, we would be grateful to hear whether our responses have addressed your main concerns. If there are any remaining questions or points that would benefit from further clarification, please let us know. We would be happy to provide additional information.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Revision Summary to Reviewers

We thank all reviewers for their constructive feedback. Four concerns were raised across the reviews. Each is answered by a new control in the revised PDF. All changed text is highlighted in blue.

  1. Does the depth gain survive the full pretraining recipe? (all reviewers) Yes. Under identical pretraining, removing only the depth branch lowers RoboCasa success from 79.2 to 74.1 (+5.1), against +4.8 without pretraining (Sec. 4.1).
  2. Does ANS gain from noise coupling or from clean-action training? (all reviewers) Both. A 2×2 control and a sweep over the clean-action probability p show that the two components of Eq. (4) contribute additively; with p matched, coupling improves all six generation metrics (Table 3(c)).
  3. Is the reconstruction comparison on a common protocol? (7NvZ, Jm6g) Sec. 4.2 now specifies the protocol, the checkpoint of every row and the camera information given to each method. With matched actions and shared calibration, the geometry gap is unchanged (new Table 2(b)).
  4. Does the gain need the interleaved design? (sdXJ) A simple auxiliary depth head leaves success at the no-depth level: 63.9, vs. 67.8 for the interleaved branch (Table 3(a)).

Further additions

  • Table 2(a): a fully pretrained X-WAM row as a reference, better on all six metrics.
  • Appendix E: pose error of the predicted end-effector states.

Claims and positioning

  • ANS: "strictly matches the test-time distribution" is corrected to a statement about support. The speed-up is attributed to asynchronous inference and the quality gain to the two components of ANS. We no longer claim a success-rate advantage for ANS.
  • Related Work positions X-WAM against 4D world models (TesserAct, Robot4DGen, MVISTA-4D), which extract actions with a separate module after generation; Sec. 4.3 relates channel concatenation to TesserAct-style fusion.

Clarifications

  • Sec. 3.2: the depth head regresses the clean VAE latent of inverse depth in one step; the depth loss reaches the shared trunk without stop-gradient.
  • Sec. 3.4 and Algorithm 2: clean-action samples (ta=0t_a = 0) carry no action or state loss.
  • Sec. 4.2 and Appendix B: the closed-loop evaluation protocol, the inverse dynamics module used to run Robot4DGen, DA3 used zero-shot, the sources and training of all baselines, and the fact that pretraining adds no benchmark trajectories beyond the fine-tuning sets.
  • Limitations: the pseudo-depth labels and the missing real-robot ablation without depth.
  • The Table 1 caption is corrected.
AuthorsOpenReview ↗

Discussion Period Summary

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their careful reading and constructive feedback, and the Area Chair for handling our submission. As the discussion period closes, we summarize where the discussion stands.

What the discussion settled. Each concern shared across the reviews is answered by a control in the revised PDF:

  • Depth under full pretraining (all reviewers): 74.1 without vs. 79.2 with the depth branch, same corpus and schedule.
  • ANS (all reviewers): coupling and clean-action training contribute additively; we narrowed the claims to support matching and generation quality.
  • Reconstruction protocol (7NvZ, Jm6g): with matched actions and calibration the geometry gap is unchanged (AbsRel −67%, CD −88%).
  • Interleaved design (sdXJ): a simple depth head stays at the no-depth level (63.9 vs. 63.0; ours 67.8).

Reviewer Jm6g confirmed that these experiments address the concerns. Reviewer sdXJ confirmed that the full-pretraining comparison "supports the usefulness of the depth branch" and that our explanations "clarify the comparison difficulties", and keeps two points open.

The two remaining points.

  1. Simple depth head under full pretraining. This needs another full pretraining run (256 H20 GPUs, 40,000 steps), not feasible during the discussion. The controls bound the answer: without pretraining the simple head gives no measurable gain, and with pretraining the interleaved branch keeps its full gain. The paper's claim, that this branch improves control under the full recipe, is tested directly.
  2. TesserAct and MVISTA-4D. TesserAct generates a single view; MVISTA-4D needs the extrinsics of every view, unknown for a moving wrist camera. Running them here would mean replacing a core component of each. The design they share, fusing depth into the video model's input, is compared with matched backbone, data and training (Table 3(a)); Robot4DGen is compared in Table 2.

We have addressed both points as far as the discussion period allows. Both ask for additional comparisons; neither disputes a reported result, and neither affects the contribution of the paper.

We are grateful for the reviewers' feedback, which has made the paper clearer and its evidence stronger, and we thank the Area Chair for considering our work.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.