- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work addresses the limited spatial understanding in current world-action models. Instead of relying on specialized prediction heads, the authors propose XGenAct, which unifies RGB observations, robot actions, metric depth, surface normals, and functional role segmentation via deterministic codecs. Consequently, XGenAct employs a single video diffusion transformer and a unified objective to learn temporal prediction, eliminating the need for modality-specific learned heads.
Critical strengths
Improving the spatial understanding of world-action models is a valuable research direction.
Critical weaknesses
- The method seems somewhat engineering-driven.
- The authors claim: We assign each functional role a fixed RGB color code shared across tasks, episodes, camera views, and frames. Please clarify how the fixed RGB color codes are assigned (e.g., via human annotation). This crucial detail is currently missing from the manuscript.
- The comparison in Table 1 is incomplete. Several important related methods are missing, including SA-WAM, RynnWorld-4D, GeoSem-WAM, DreamWAM, WAM4D, and X-WAM. In particular, SA-WAM is the closest to the proposed method and should be included.
- The authors repeatedly argue that multiple geometric and semantic prediction tasks should be expressed within a unified generative space, unlike other related methods. However, the specific benefits of this unification are not sufficiently analyzed. A deeper discussion, supported by experimental verification (e.g., ablation studies), is needed.
Questions for authors
please see weaknesses.
6 comments
W3: Comparisons with SA-WAM, RynnWorld-4D, GeoSem-WAM, DreamWAM, WAM4D and X-WAM
Appendix L now discusses these works, and Related Work points to it. In summary:
| Work | Structured signals | How they are used | Modality-specific modules | Test-time input |
|---|---|---|---|---|
| SA-WAM | metric depth | generated jointly with RGB and actions (frozen VAE) | none | multi-view RGB-D |
| RynnWorld-4D | depth, optical flow | generated as videos, jointly with RGB | per-modality branches | RGB-D |
| X-WAM | depth | dedicated depth branch | depth branch | multi-view RGB |
| GeoSem-WAM | depth, semantics | auxiliary heads, removed at deployment | DPT-style heads | RGB |
| WAM4D | future depth | auxiliary head on register tokens, removed at inference | registers, depth blocks, head | multi-view RGB, action history |
| DreamWAM | flow; depth and semantic features | training only; gated residual branches | branches, heads, flow channels | RGB |
| XGenAct | depth, normals, functional roles | generated in the shared video space via sampled templates | none (one shared head) | RGB, cameras, pose |
These works form two families: models that generate depth in a fixed joint layout (SA-WAM, RynnWorld-4D, X-WAM) and models that use geometry or semantics as training-time auxiliary targets (GeoSem-WAM, WAM4D, DreamWAM). XGenAct extends the first family: it samples single-perception templates instead of a fixed layout, needs no modality-specific branch, covers normals and functional roles in addition to depth, and decodes actions from images. Our auxiliary-head variant reproduces the second family's design inside our backbone with everything else fixed, so the 19/20 vs. 13/20 result directly compares the two families' core choices.
Empirical comparison. None of these works reports RLBench results, and at the time of writing SA-WAM's code was "coming soon" and no usable code was available for GeoSem-WAM or WAM4D. We therefore evaluate both design families on the real robot, either inside our backbone or with released systems fine-tuned on the same 42 demonstrations (Table 3; protocol, confidence intervals and failure types in our response to nfba W3):
| Model on the real robot | Relation to the named works | Success |
|---|---|---|
| XGenAct | geometry generated in the shared space | 19/20 |
| Auxiliary-head variant | auxiliary-target design of GeoSem-WAM, WAM4D, DreamWAM; same backbone, labels and recipe as XGenAct | 13/20 |
| Cosmos3-Edge (RGB) | Cosmos 3 world action model, no geometry | 6/20 |
| SA-WAM-style | SA-WAM's depth encoding (depth generated jointly with RGB) on Cosmos3-Edge | 4/20 |
| Fast-WAM | Wan2.2-5B world action model, code base of DreamWAM | 3/20 |
| π0.5 | strongest policy of Table 1 | 9/20 |
XGenAct outperforms every counterpart. Within one backbone, generating the geometry beats the auxiliary-target family on identical labels (19/20 vs. 13/20), and adding generated depth in the manner of SA-WAM does not improve Cosmos3-Edge (4/20 vs. 6/20). These are controlled counterparts and released systems rather than reproductions of the published models, so they compare design choices, not the original implementations.
W4: Specific benefits of unification
The matched-supervision ablation measures exactly this (General Response; §4.4). With the same labels, data and action supervision, generating the geometry in the shared video space (19/20) outperforms regressing it as conventional auxiliary targets (13/20, p = 0.04) and generating it through private output heads (9/20, p = 0.001). Appendix M identifies the reason: only unified generation aligns geometry learning with action learning (Table 18).
Unification brings further measurable benefits that auxiliary heads lack:
- Decodable structured outputs. One checkpoint outputs metric depth, normals and role maps through the same decoder, and direct generation beats RGB→frozen-expert pipelines for depth and segmentation without filtering, on 13/14 and 14/14 of 14 held-out tasks (Tables 9–10).
- Extensibility. A new structured task enters through a parameter-free codec with a measured round-trip admission test (Tables 5 and 17), without new modules.
- Constant query cost. Because every template is a 4-stream canvas, adding modalities does not slow down an action query (Table 16).
Its operating range is also documented: a query takes 48 s at 20 denoising steps on an RTX 5090 and returns a full 41-waypoint plan, which suits tasks that tolerate planning pauses, and separately prompted modalities are not used as a single joint future (Appendix K).
We sincerely appreciate the reviewer's insightful comments and clear list of concerns.
The revision tests each design choice directly and adds a mechanistic explanation of why the design works:
Mechanism: why generating the labels helps more than regressing them (Appendix M). Both models are analysed under the deployment prompt (current RGB frames and pose only).
XGenAct's advantage therefore lies in how geometry supervision is learned: generated in the shared video space, every geometry update also advances the action; regressed as an auxiliary target, it does not. This matches the 19/20 vs. 13/20 result under identical labels.
Thank you for pointing out this missing detail; Appendix C now documents it. Role maps are produced in three steps:
The role → RGB mapping is a fixed 9-entry palette shared across tasks, episodes, views and frames (8 training roles plus a reserved unknown colour). An object not covered by its task table is encoded as unknown, and the data validator rejects any tree containing it; the training and evaluation trees contain no unknown pixels. Adding a task thus costs one small semantic table rather than per-pixel annotation; the paper now states this explicitly.
No privileged information at test time. Role maps, depth and normals are training targets only. Closed-loop control (Tables 1–3) prompts
video+actionand conditions only on the current RGB frames and the end-effector pose. On the real robot, arm and gripper masks are rendered from the measured joint angles, cubes and box are tracked with SAM 2, and the target is the cube nearest the gripper at the closing frame (Appendix J.1).