- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
X-WAM is a unified world-action model fine-tuned from a video DiT (Wan2.2-TI2V-5B). It consumes multi-view RGB, proprioceptive state, and language, and jointly predicts future RGB videos, depth videos, states, and actions. Two designs are central. First, a lightweight depth branch replicates the final M DiT blocks and reads the main branch through "unilateral attention" (depth attends to the main branch, and not vice versa); because the branch can be switched off at inference, action latency is unaffected. Second, Asynchronous Noise Sampling (ANS) couples the training noise levels of video and action within a joint distribution and adopts asynchronous denoising (actions in T_a steps, video in T_O), allowing actions to be dispatched early. The model is pretrained on roughly 5,874 hours of data and evaluated on RoboCasa, RoboTwin 2.0, 4D reconstruction/generation metrics, ablations, and real-robot tasks.
Critical strengths
- A practical, latency-aware design. Decoupling a depth-supervised branch from the action path (so it can be disabled at inference) is a concrete answer to the "quality vs. real-time" tension, and the ablation in Table 3 backs the claimed latency/quality trade-off.
- A coupled training/inference noise distribution. Framing the asynchronous schedule as a coupled sampling problem, and showing that decoupled-Async degrades reconstruction (Table 3b), is a clean and well-motivated contribution.
Critical weaknesses
- The reconstruction comparison needs a more explicit common protocol. Table 2 reports strong RGB-D and point-cloud gains, but Section 4.2 does not make sufficiently clear whether methods are compared on the same action-conditioned future and how camera information is supplied. These details matter more to interpreting the reported gains than simply adding another benchmark; the current quantitative evidence is also confined to RoboCasa.
- The control benefit under the full training recipe remains incompletely isolated. Table 3 supports a benefit from depth supervision without large-scale pretraining, whereas the real-robot comparison supports competitive performance rather than a causal improvement from depth modeling. The corresponding contribution after full pretraining is still not cleanly isolated.
Questions for authors
- Reconstruction protocol. For Table 2, are all methods compared on the same action-conditioned future, or on each policy's own rollout? A matched-action comparison with shared calibration would isolate the geometry gains.
- ANS: coupling vs. clean actions. Can the independent-sampling baseline in Table 3(b) receive the same clean-action probability p at fixed inference steps? This separates noise coupling from merely covering the late denoising regime.
- Depth under full pretraining. Please add a depth/no-depth control under the full pretraining recipe, and clarify which shared parameters receive depth-loss gradients.
3 comments
Thank you for the detailed rebuttal and additional experiments. The full-experiments address my concerns. I will therefore keep my positive score.
We sincerely thank the reviewer for their thoughtful feedback and positive assessment of our work. We greatly appreciate your time and constructive suggestions throughout the discussion.
Response to Reviewer Jm6g
Thank you for the focused and constructive review. Short answers to the three questions come first; details follow. All new results are in the revised PDF.
W1 / Q1. A common protocol for the reconstruction comparison.
Same future or own rollout. Each method was evaluated on its own closed-loop rollout: it executed its own predicted actions, and its predictions were compared with the frames the simulator rendered along that rollout. Robot4DGen does not provide code for its action pipeline, so we ran it through an external inverse dynamics module that predicts actions from its generated frames. We omitted this from the submission and apologize; Sec. 4.2 now describes the protocol.
Matched actions, shared calibration. The baselines cannot be conditioned on given actions, so the comparison the reviewer suggests is possible between X-WAM and X-WAM w/o depth + DA3. On 2,400 held-out clips, both start from the same frame, receive the same ground-truth actions, and use the same ground-truth calibration, including the wrist pose:
AbsRel falls by 67% and CD by 88%, the same relative gaps as under the original protocol (0.1045 → 0.0349 and 0.0401 → 0.0049). The geometry gain therefore comes neither from different trajectories nor from calibration. What remains is the depth branch itself, trained with in-domain depth, against a zero-shot estimator; architectures under equal depth supervision are compared in Table 3(a).
Camera information (now in Sec. 4.2).
W2 / Q3. Depth under the full training recipe; gradient path.
Alongside the full X-WAM, we had also pretrained a variant without the depth branch, with the same 5,874-hour corpus, schedule and hyperparameters. Fine-tuned on RoboCasa as in Table 1:
The depth gain has the same size with and without pretraining.
Which parameters receive depth gradients. The depth loss is back-propagated without stop-gradient. It updates the M depth blocks and the depth head and, through the cross-attention of Eq. (3), the shared trunk (blocks 1 to N−M) and main-branch blocks N−M+1 to N−1. The last main block and the action, state and video heads receive no depth gradient. The main branch's forward pass never reads depth tokens, so the branch can be switched off at inference.
Q2. ANS: coupling vs. clean actions.
We crossed noise coupling with p at fixed inference steps (Ta=5, TO=25; each cell is SR / PSNR / AbsRel / CD):
With p matched, coupling improves all six reconstruction metrics at either value (at p = 0 / 0.5: PSNR +0.35 / +0.45 dB, AbsRel −6.5% / −5.9%, CD −6.6% / −5.8%). Clean-action exposure contributes as well (AbsRel and CD about −14% under either scheme), and the two effects are additive. Both are parts of Eq. (4), its second and first case, and correspond to the two phases of asynchronous inference. The four success rates are close (67.2–67.8): the benefit of ANS is generation quality, at the 4.5× lower action latency of asynchronous inference.
In sum, the geometry gain is now isolated from trajectories and calibration (Table 2(b)), and the control gain from pretraining (Sec. 4.1). We are glad to clarify anything further.