acceptodds
Under review as a conference paper at ICLR 2027

XGenAct: Geometry-Enhanced World Action Models through Cross-Task Generation

Abstract

World action models (WAMs) have advanced robot control by predicting how observations and actions evolve over time. Despite this progress, RGB and action based future prediction does not explicitly address the spatial understanding needed for robot manipulation. Existing efforts often add a limited set of spatial prediction tasks through specialized heads or branches, leaving both the range of spatial supervision and the model architecture fragmented. We introduce XGenAct, a world action model that represents RGB observations, robot actions, metric depth, surface normals, and functional role segmentation as RGB videos through deterministic codecs. By sampling perception and action tasks during training, XGenAct uses one video diffusion transformer and one objective to learn temporal prediction across these spaces without modality specific learned heads. On held out RLBench tasks, structured perception training improves average closed loop success over RGB only training, and XGenAct achieves 52% success in the five task external comparison, versus 26% for the strongest evaluated baselines. It also predicts future depth and segmentation more accurately than the evaluated pipelines that generate RGB first and then apply a frozen perception expert.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action models, robotic manipulation, multimodal generation, video diffusion, embodied perception, unified representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hLCNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses the limited spatial understanding in current world-action models. Instead of relying on specialized prediction heads, the authors propose XGenAct, which unifies RGB observations, robot actions, metric depth, surface normals, and functional role segmentation via deterministic codecs. Consequently, XGenAct employs a single video diffusion transformer and a unified objective to learn temporal prediction, eliminating the need for modality-specific learned heads.

Critical strengths

Improving the spatial understanding of world-action models is a valuable research direction.

Critical weaknesses
  1. The method seems somewhat engineering-driven.
  2. The authors claim: We assign each functional role a fixed RGB color code shared across tasks, episodes, camera views, and frames. Please clarify how the fixed RGB color codes are assigned (e.g., via human annotation). This crucial detail is currently missing from the manuscript.
  3. The comparison in Table 1 is incomplete. Several important related methods are missing, including SA-WAM, RynnWorld-4D, GeoSem-WAM, DreamWAM, WAM4D, and X-WAM. In particular, SA-WAM is the closest to the proposed method and should be included.
  4. The authors repeatedly argue that multiple geometric and semantic prediction tasks should be expressed within a unified generative space, unlike other related methods. However, the specific benefits of this unification are not sufficiently analyzed. A deeper discussion, supported by experimental verification (e.g., ablation studies), is needed.
Questions for authors

please see weaknesses.

6 comments

We sincerely appreciate the reviewer's insightful comments and clear list of concerns.

W1: Is the method engineering-driven?

The revision tests each design choice directly and adds a mechanistic explanation of why the design works:

  • Generated geometry vs. conventional auxiliary supervision with the same labels: 19/20 vs. 13/20 on the real robot (p = 0.04; §4.4).
  • Single shared output head vs. private heads: 19/20 vs. 9/20 (p = 0.001), although the private-head variant starts from the same function.
  • Structured supervision vs. RGB-only: 32.5% → 52.5–61.25% in simulation (Table 2); on the real robot XGenAct reaches 19/20 against 3/20–9/20 for π0.5 and three world action models trained on the same demonstrations (Table 3).
  • Direct generation vs. generating RGB and post-processing it: better for depth and segmentation without filtering, on all four original tasks and on 13/14 and 14/14 of 14 held-out tasks; on par for normals (Tables 9–10).
  • Codec design and action fidelity: our depth code is the most accurate after the frozen VAE (Table 17), which adds only 0.1–0.5 mm and about 1° to the action codec's error (Table 5).
  • Cost: single pairing makes an action query 3.4× cheaper than a joint 10-stream layout (Appendix K).

Mechanism: why generating the labels helps more than regressing them (Appendix M). Both models are analysed under the deployment prompt (current RGB frames and pose only).

  • Geometry training transfers to the action only when geometry is generated. On 40 windows, the cosine between the deployment action-loss gradient and the geometry-objective gradient on the shared blocks (reliability-corrected) is +0.12 / +0.09 / +0.43 (depth / normals / roles) for XGenAct, but −0.00 / +0.02 / +0.01 for auxiliary regression. The alignment is acquired through shared generation: the same generative objective gives +0.00 / +0.01 / +0.11 at initialization and +0.01 / +0.02 / +0.05 on the auxiliary-head weights (Table 18).
  • Both policies read internal geometry. Probes on the action tokens recover the target cube's position in both models (R² 0.97–0.98 from block 14 on, vs. 0.83–0.85 at initialization), and both plans depend causally on it: deleting the instructed cube shifts the plan by about 10 cm versus under 1 cm for background, and erasing the geometry subspace shifts it about 4 cm more than a random subspace (Table 19).

XGenAct's advantage therefore lies in how geometry supervision is learned: generated in the shared video space, every geometry update also advances the action; regressed as an auxiliary target, it does not. This matches the 19/20 vs. 13/20 result under identical labels.

W2: How functional-role colour codes are assigned

Thank you for pointing out this missing detail; Appendix C now documents it. Role maps are produced in three steps:

  1. Pixel → object instance comes entirely from the simulator's instance masks (RLBench handle maps, ManiSkill3 segmentation ids); no pixel is hand-labelled.
  2. Object instance → role is a small table per task (3–15 entries) mapping simulator object names to a base role, plus global rules for robot links, gripper fingers and static scene elements. In close jar the jar bodies are goal and the lid distractor; in sweep to dustpan the dustpan is goal, the broom tool, its holder fixture and the dirt distractor. Identical geometry can take different roles across tasks (drawer boxes are manipulated objects in open drawer but destinations in put item in drawer), because the role encodes how a region participates in the task.
  3. Instruction → target: at load time, the object group named by the instruction is promoted from distractor to target.

The role → RGB mapping is a fixed 9-entry palette shared across tasks, episodes, views and frames (8 training roles plus a reserved unknown colour). An object not covered by its task table is encoded as unknown, and the data validator rejects any tree containing it; the training and evaluation trees contain no unknown pixels. Adding a task thus costs one small semantic table rather than per-pixel annotation; the paper now states this explicitly.

No privileged information at test time. Role maps, depth and normals are training targets only. Closed-loop control (Tables 1–3) prompts video+action and conditions only on the current RGB frames and the end-effector pose. On the real robot, arm and gripper masks are rendered from the measured joint angles, cubes and box are tracked with SAM 2, and the target is the cube nearest the gripper at the closing frame (Appendix J.1).

W3: Comparisons with SA-WAM, RynnWorld-4D, GeoSem-WAM, DreamWAM, WAM4D and X-WAM

Appendix L now discusses these works, and Related Work points to it. In summary:

WorkStructured signalsHow they are usedModality-specific modulesTest-time input
SA-WAMmetric depthgenerated jointly with RGB and actions (frozen VAE)nonemulti-view RGB-D
RynnWorld-4Ddepth, optical flowgenerated as videos, jointly with RGBper-modality branchesRGB-D
X-WAMdepthdedicated depth branchdepth branchmulti-view RGB
GeoSem-WAMdepth, semanticsauxiliary heads, removed at deploymentDPT-style headsRGB
WAM4Dfuture depthauxiliary head on register tokens, removed at inferenceregisters, depth blocks, headmulti-view RGB, action history
DreamWAMflow; depth and semantic featurestraining only; gated residual branchesbranches, heads, flow channelsRGB
XGenActdepth, normals, functional rolesgenerated in the shared video space via sampled templatesnone (one shared head)RGB, cameras, pose

These works form two families: models that generate depth in a fixed joint layout (SA-WAM, RynnWorld-4D, X-WAM) and models that use geometry or semantics as training-time auxiliary targets (GeoSem-WAM, WAM4D, DreamWAM). XGenAct extends the first family: it samples single-perception templates instead of a fixed layout, needs no modality-specific branch, covers normals and functional roles in addition to depth, and decodes actions from images. Our auxiliary-head variant reproduces the second family's design inside our backbone with everything else fixed, so the 19/20 vs. 13/20 result directly compares the two families' core choices.

Empirical comparison. None of these works reports RLBench results, and at the time of writing SA-WAM's code was "coming soon" and no usable code was available for GeoSem-WAM or WAM4D. We therefore evaluate both design families on the real robot, either inside our backbone or with released systems fine-tuned on the same 42 demonstrations (Table 3; protocol, confidence intervals and failure types in our response to nfba W3):

Model on the real robotRelation to the named worksSuccess
XGenActgeometry generated in the shared space19/20
Auxiliary-head variantauxiliary-target design of GeoSem-WAM, WAM4D, DreamWAM; same backbone, labels and recipe as XGenAct13/20
Cosmos3-Edge (RGB)Cosmos 3 world action model, no geometry6/20
SA-WAM-styleSA-WAM's depth encoding (depth generated jointly with RGB) on Cosmos3-Edge4/20
Fast-WAMWan2.2-5B world action model, code base of DreamWAM3/20
π0.5strongest policy of Table 19/20

XGenAct outperforms every counterpart. Within one backbone, generating the geometry beats the auxiliary-target family on identical labels (19/20 vs. 13/20), and adding generated depth in the manner of SA-WAM does not improve Cosmos3-Edge (4/20 vs. 6/20). These are controlled counterparts and released systems rather than reproductions of the published models, so they compare design choices, not the original implementations.

W4: Specific benefits of unification

The matched-supervision ablation measures exactly this (General Response; §4.4). With the same labels, data and action supervision, generating the geometry in the shared video space (19/20) outperforms regressing it as conventional auxiliary targets (13/20, p = 0.04) and generating it through private output heads (9/20, p = 0.001). Appendix M identifies the reason: only unified generation aligns geometry learning with action learning (Table 18).

Unification brings further measurable benefits that auxiliary heads lack:

  • Decodable structured outputs. One checkpoint outputs metric depth, normals and role maps through the same decoder, and direct generation beats RGB→frozen-expert pipelines for depth and segmentation without filtering, on 13/14 and 14/14 of 14 held-out tasks (Tables 9–10).
  • Extensibility. A new structured task enters through a parameter-free codec with a measured round-trip admission test (Tables 5 and 17), without new modules.
  • Constant query cost. Because every template is a 4-stream canvas, adding modalities does not slow down an action query (Table 16).

Its operating range is also documented: a query takes 48 s at 20 denoising steps on an RTX 5090 and returns a full 41-waypoint plan, which suits tasks that tolerate planning pauses, and separately prompted modalities are not used as a single joint future (Appendix K).

Reviewer_jWPjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

XGenAct represents RGB observations, metric depth, surface normals, functional-role segmentation, and robot actions as videos through deterministic codecs. These representations share a frozen VAE, a video diffusion Transformer, and a training objective. Each training example pairs one perception modality with actions, allowing geometric and semantic supervision to influence a shared action generator. Experiments on RLBench show improved average control success with structured perception training, including 52% success in the external comparison. The paper also reports that directly generating depth and segmentation outperforms generating RGB followed by applying frozen perception experts.

Critical strengths
  1. The unified representation is simple and extensible. Multiple perception signals and robot actions share a common generation architecture.
  2. The internal ablation supports structured supervision. Table 2 compares eight modality menus under matched backbone, optimization settings, and action-supervision budgets, improving average success from 32.5% to 52.5–61.25%.
  3. Implementation details are well documented. The appendix describes data construction, modality codecs, control interfaces, and evaluation protocols.
  4. The discussion acknowledges modality trade-offs. Section 4.3 recognizes that adding more modalities does not always improve performance and describes the task-level interpretations as correlational.
Critical weaknesses
  1. The contribution beyond additional supervision is unclear. The ablations demonstrate benefits from adding structured modalities but do not isolate the value of the shared generative formulation.
  2. Single-modality pairing is insufficiently justified. The paper does not adequately explain why each generation pairs only one perception modality with actions, rather than jointly generating complementary modalities, or discuss the implications for computational cost and cross-modal consistency.
  3. The “no manual balancing” claim needs qualification. Modality-sampling probabilities still control task contributions, while different latent error scales may affect optimization.
  4. Perception averages cover a restricted task subset. The reported averages retain only two tasks for RGB, depth, and normals, and three for segmentation. This limits how broadly the results in Figure 4 can be interpreted.
  5. The execution-gap filtering needs stronger justification. For each metric, a task is excluded from all routes’ averages if any listed configuration has a median commanded–achieved end-effector position gap of at least 10 mm. The authors should justify this threshold and clarify whether the conclusions hold without filtering.
Questions for authors
  1. What evidence distinguishes the benefit of the shared generative formulation from that of additional structured supervision?
  2. Why was single-perception-modality pairing chosen over joint generation of multiple perception modalities?
  3. How were the modality-sampling probabilities selected, and how should the claim of avoiding manual balancing be interpreted?
  4. Why was the 10 mm exclusion threshold chosen? Using existing results, do the main perception conclusions hold when all tasks are included?

Overall, the architecture is appealing, and the internal ablation demonstrates useful control improvements. However, the contribution beyond additional supervision and the rationale for single-modality pairing remain insufficiently established. The filtered perception results also support a narrower conclusion than the full evaluation suite suggests.

6 comments

We thank the reviewer for the detailed and insightful comments.

W1 + Q1: Shared generative formulation vs. additional supervision

The new matched-supervision ablation separates the two (General Response; §4.4). Three models are trained from the same checkpoint with the same data, labels, recipe and action supervision. On the real robot (unseen target colours), XGenAct succeeds in 19/20 trials; the same labels regressed by conventional auxiliary heads give 13/20 (p = 0.04), and private output heads give 9/20 (p = 0.001). Together with Table 2, this isolates three effects: (i) structured supervision at all (Table 2: 32.5% → 52.5–61.25%); (ii) generating it in the shared space rather than regressing it (19/20 vs. 13/20); (iii) sharing the output head (19/20 vs. 9/20). Appendix M shows the mechanism: only shared generation aligns geometry gradients with the action gradient.

W2 + Q2: Why single-perception pairing rather than joint generation?

XGenAct combines complementary modalities through its shared generator rather than a shared canvas: every template trains the same transformer and output head, so depth, normals and roles all shape the weights that generate the action. This makes single pairing more effective per unit of compute and more flexible than joint generation (Appendix K, M):

  • Complementarity is realized in the shared weights. Combining modalities across samples works: the menu with normals and roles reaches 61.25% average success, the best in Table 2. After training, each single template produces geometry gradients aligned with the deployment action gradient (+0.12 / +0.09 / +0.43 for depth / normals / roles; Table 18), so every sample transfers its geometry to the policy without placing all modalities on one canvas.
  • Joint generation did not help on the real robot. The one model that generates depth jointly with RGB, SA-WAM-style, does not improve on its RGB-only counterpart (4/20 vs. 6/20), whereas XGenAct with single pairing reaches 19/20 (Table 3).
  • Action queries are 3.4× cheaper. Deployment needs only RGB+action; a joint canvas would make every action query pay for all modalities, turning a 44 s real-robot query into about 2.2 min and a 6 h evaluation of Table 2 into 18 h (Table 16):
Layout (two views)StreamsTokensOne denoiser passRelativePeak memory
X + action (ours)411,2640.91 s1.0×14.2 GB
X + Y + action616,8961.52 s1.7×14.7 GB
RGB + D + N + S + action1028,1603.07 s3.4×18.6 GB
  • Training stays feasible and the menu stays open. The 4-stream model already uses 77 of 80 GB on one H100, and a 10-stream canvas is 2.5× longer. With single pairing, a new modality enters as a parameter-free codec, one template and a sampling probability, validated before training (Tables 5, 17), and the cost per query does not change.

Scope. Single pairing gives up only synchronized multi-modal output, which control does not use (deployment issues one RGB+action query). When several synchronized modalities are needed, for example for 3D reconstruction, the same model already accepts joint canvases (the layouts profiled above), so joint training is a direct extension (Appendix K).

W3 + Q3: Sampling probabilities and the "no manual balancing" claim

We agree that the template probabilities control task exposure and have revised the claim accordingly (Introduction, §3.4): "XGenAct uses one flow-matching objective without modality-specific output heads or hand-tuned per-modality loss coefficients; the template-sampling distribution is a training choice."

How the probabilities were selected. The menu follows two fixed principles rather than a search: RGB, the deployment input, receives the largest share (0.4), and the three geometric streams receive equal exposure (0.2 each). The same menu is used in simulation and on the real robot, and it was never tuned on the evaluation tasks.

How to interpret the claim. Instead of a loss weight and a head per modality, as in auxiliary-target designs, XGenAct has a single, directly interpretable control: the fraction of training samples given to each template. As the reviewer notes, equal sampling does not imply equal optimization weight, because latent error scales differ. We now measure this (Appendix D): under the same menu, the perception loss splits 42/18/21/18% (RGB/depth/normals/roles) in simulation and 36/26/31/7% on the real robot. This loss share is a practical diagnostic for adjusting the menu to a new domain.

Robustness to the ratio (Appendix D). Two models that differ only in the menu, 0.4/0.2/0.2/0.2 and uniform 0.25×4, reach 37/80 (46.2%, 95% CI 35.7–57.1) and 34/80 (42.5%, 32.3–53.4) on the four Table 2 tasks (20 paired scenes per task; exact McNemar p = 0.66). Performance is stable between the two menus, so the results do not hinge on a finely tuned ratio.

W4: Perception results on all tasks, without filtering

We thank the reviewer for this careful observation. The main-text perception result (Figure 5, §4.5) now uses no filtering: it averages all four tasks for every training menu, and the filtered averages remain only in the per-task Table 8 for reference. We also extend the evaluation from four to 14 held-out RLBench tasks (Appendix I, new Table 10). In both, XGenAct's direct generation is more accurate than generating RGB and applying a frozen expert.

Four original tasks, every training menu, no filtering (Figure 5; values in Table 9):

Modality (metric)MenuXGenAct (direct)Generated RGB → frozen expertXGenAct − expert
Depth (AbsRel ↓)RGB+D0.3900.485−0.095
RGB+D+S0.3950.468−0.073
RGB+D+N0.3130.463−0.150
RGB+D+N+S0.2680.458−0.190
Segmentation (mIoU ↑)RGB+S0.4350.220+0.215
RGB+D+S0.3880.225+0.163
RGB+N+S0.4100.200+0.210
RGB+D+N+S0.3650.248+0.118
Normals (cosine ↑)RGB+N0.7300.735−0.005
RGB+N+S0.7280.740−0.013
RGB+D+N0.7130.745−0.033
RGB+D+N+S0.7230.750−0.028

Every menu shows the same pattern: direct generation wins for depth and roughly doubles segmentation mIoU, while normals are on par. Future RGB is retained: the perception menus reach LPIPS 0.42–0.47, versus 0.46 for RGB-only training.

14 held-out tasks, no filtering (Table 10; the four original tasks plus ten more, five trials per task, RGB+D+N+S model, same experts):

14 held-out tasks, no filteringXGenAct (direct)Generated RGB → frozen expertXGenAct − expert (95% CI)XGenAct better on
Depth (AbsRel ↓)0.2880.415−0.127 (−0.167 to −0.088)13 / 14 tasks
Segmentation (mIoU ↑)0.3820.251+0.131 (+0.098 to +0.168)14 / 14 tasks

Both intervals exclude zero. XGenAct's direct generation is therefore the more accurate route to future geometry and semantics, independent of task selection and filtering.

W5 + Q4: The 10 mm execution-gap threshold

Why a filter is needed. Perception is scored against the simulator state reached by executing the model's own decoded actions. When the controller cannot follow the commanded poses, because of inverse-kinematics limits or contact (mainly on toilet seat down and meat on grill), the generated frame shows the commanded arm pose while the simulator shows a different one. The error then measures execution failure rather than perception quality, and the gate in the submission removed such tasks.

Why 10 mm, and why it cannot favour our method. The threshold is about twice the median decoding error of the action codec (4.9 mm, Table 5), so larger gaps reflect execution rather than decoding noise. It is computed only from commanded and achieved simulator poses, never from the predictions, and it removes a task from every route's average at once, so it cannot favour our direct generation over the RGB→expert route. It was not tuned on the results.

The revised results no longer depend on it. The main-text Figure 5 now reports unfiltered means over all four tasks: direct generation remains better for depth in every menu (AbsRel 0.31–0.40 vs. 0.46–0.49) and roughly doubles segmentation mIoU (0.39–0.44 vs. 0.20–0.23), with normals on par (Figure 5, Table 9). On 14 held-out tasks the gap stays below 1.3 mm on every task and route except one cell, so no task is filtered, and direct generation is more accurate for depth on 13/14 tasks and for segmentation on 14/14 (W4, Table 10). The gated averages are kept only in Table 8 for reference, where they lead to the same conclusions.

Reviewer_nfbaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

XGenAct extends Action Images with depth, normals, and functional-role segmentation encoded as RGB videos. A shared video backbone learns sampled perception–action tasks, improving held-out RLBench control over RGB-only training.

Critical strengths
  1. The shared representation avoids additional learned modality-specific output heads and offers a simple interface for adding structured perception targets.

  2. Matched internal training comparisons show substantial gains from structured perception supervision. Codec tests and detailed baseline descriptions improve transparency.

Critical weaknesses
  1. Appendix H explicitly states that Evo-1’s rotation-target conversion is erroneous; camera conditioning and execution budgets also differ across methods. This limits Table 1 as evidence of broad superiority, although it does not invalidate the internal RGB-only comparison.

  2. The architecture and action representation largely follow Action Images, while experiments do not isolate shared video encoding from alternative ways of incorporating the same geometric targets.

  3. Although training uses data from both RLBench and ManiSkill3, closed-loop policy evaluation is conducted only in RLBench, with no real-robot and other benchmark experiments like RoboDojo or Robotwin C2R.

  4. There are too many papers that introduce multi-modal information to benefit WAM. S-VAM already exploits geometric and semantic foresight, while Cosmos Policy jointly models actions, future observations, and values within an unchanged video-diffusion backbone. Since XGenAct also inherits its action representation from Action Images, the paper should clarify what substantive advance its image-space codecs and cross-task training provide beyond extending established unified-generation approaches with additional supervision targets.

Questions for authors
  1. How does the external comparison change when the defective Evo-1 result is excluded, and which conclusions survive the interface differences?

  2. What evidence attributes the control gains specifically to the shared generative representation?

  3. What are the action-decoding error and complete per-query latency, including generation, VAE decoding, and pose recovery?

10 comments

We thank the reviewer for the detailed and insightful comments.

W1 + Q1: Table 1 without Evo-1

We appreciate the reviewer catching this. Our conversion of RLBench rotations into Evo-1's per-step delta format subtracted axis-angle vectors, which wrap at ±π, so about 2% of its training targets were corrupted. Since the resulting number did not reflect Evo-1 itself, we have removed it from Table 1 and from the paper. XGenAct's lead is unchanged: it averages 52%, twice the strongest remaining baselines (π0.5 and VLANeXt, 26%), is best on 3 of 5 tasks, and is the only method above 10% on every task. To make the remaining differences in interfaces transparent, Table 7 now gives a protocol matrix (camera input, resolution, predicted/executed chunk, action target, training exposure, termination rule, harness), and the real-robot comparison in W3 adds a second evaluation with a shared robot, training data and success criterion.

W2 + Q2: Isolating the shared generative representation

The new matched-supervision ablation (General Response; §4.4; App. J) holds initialization, data, labels, recipe and action supervision fixed and varies only how geometry enters training. XGenAct succeeds in 19/20 real-robot trials, against 13/20 when the same labels are regressed by auxiliary heads (p = 0.04) and 9/20 when they are generated through private output heads (p = 0.001).

  • Generation vs. auxiliary regression: with identical labels, generating them in the shared video space is significantly more effective (p = 0.04), so the gain is not explained by the extra labels.
  • Shared vs. private heads: the private-head variant starts from XGenAct's exact function with only 0.037% more parameters, yet reaches 9/20; a single shared head is the better design.
  • Why: only shared generation produces geometry gradients aligned with the action gradient (Appendix M, Table 18; W4).

Recipe (App. J.2, Table 12). All three models: Action Images checkpoint (step 125,750), 5,000 steps, effective batch 4, learning rate 2e-6 with 200 warm-up steps, seed 42, template menu 0.4/0.2/0.2/0.2. Two real-robot choices apply to all three: each training window starts at a random frame (the first frame with probability 0.25) and covers the remainder of the demonstration, resampled to 41 frames, so one query plans to the end of the task; and the deployment plan (current frame only) is sampled for 50% of action-bearing examples.

W3: Real-robot evaluation

XGenAct succeeds in 19 of 20 real-robot trials with target colours it never saw as instructions, more than twice the success of the best external policy trained on the same demonstrations (π0.5, 9/20), while using eight times fewer training samples (§4.4, Table 3, Figure 4; details in Appendix J; videos of all trials in the supplementary material).

PolicyInitializationSuccess95% CIp vs. XGenActMost frequent failure
XGenActAction Images19/2076–99%—missed grasp (1)
π0.5 (top + side)π0.5 base9/2026–66%0.001missed grasp (7)
Cosmos3-Edge (RGB)Cosmos3-Edge6/2015–52%4×10⁻⁵no approach (7), missed grasp (6)
SA-WAM-style (Cosmos3-Edge, RGB-D)Cosmos3-Edge4/208–42%2×10⁻⁶no approach (15)
Fast-WAMFast-WAM base (Wan2.2-5B)3/205–36%4×10⁻⁷missed grasp (13)

The baselines mostly fail before lifting the cube, by missing the grasp or never approaching the instructed cube; XGenAct's single failure is a near-miss grasp. Each query of XGenAct returns a 41-waypoint plan to the end of the task: 14 of its 19 successes are completed from one plan, the other five with one re-plan (Figure 4 shows one of each). Adding generated depth to Cosmos3-Edge in the manner of SA-WAM does not improve it (4/20 vs. 6/20).

Setup. UFactory xArm6, parallel gripper, two fixed RealSense D435 cameras (top, side). Instruction: "pick up the ⟨colour⟩ cube and place it into the box", with three to six cubes on the table. Training: 42 teleoperated demonstrations (20 Hz, 6.8–13.0 s) whose instructions name blue, yellow, red or green; every evaluation trial instructs an orange or purple cube, colours seen only as distractors. Depth, normal and role targets are produced automatically (D435 depth; Marigold normals aligned to the calibrated table plane; role maps from robot links rendered at the measured joint angles plus SAM 2 tracks). At test time every policy receives only the two current RGB frames and the end-effector pose.

Protocol. All baselines are fine-tuned on the same 42 demonstrations with 1.6×10⁵ samples each (XGenAct: 2×10⁴), from their released checkpoints and with their own action interfaces (Table 13). They are re-queried after every chunk within a 26 s budget (twice the longest demonstration); XGenAct uses 20 denoising steps and at most two queries. All methods share workspace clipping, minimum tool height and speed limit; 20 trials per method.

Choice of baselines. We chose policies with released code that trains directly on our demonstrations (all four run through LeRobot), covering the strongest policy of Table 1 (π0.5) and the world action models closest to the works the reviewers named (Fast-WAM, the code base of DreamWAM; Cosmos3-Edge; SA-WAM-style, our re-implementation of SA-WAM's depth encoding, since SA-WAM has no released code).

Coverage beyond RLBench. We prioritized the real robot, the first setting the reviewer listed, as the strongest external test of the method, and broadened the simulated evaluation: the perception study now covers 14 held-out RLBench tasks instead of four (Table 10; jWPj W4).

W4: What image-space codecs and cross-task training add

The action representation is inherited from Action Images. XGenAct contributes two things on top of it, and the revision measures both (Appendix M): (i) codecs that carry metric geometry and functional roles intact through a frozen video tokenizer, and (ii) cross-task generation, in which learning to generate geometry trains the same pathway that generates actions.

(i) Codecs designed for the frozen tokenizer (Table 17). The frozen VAE blurs and shifts colours, so a useful code must degrade gracefully under exactly these errors. Round-tripping alternative depth codes through the frozen Wan2.2 VAE (24 real-robot and 20 RLBench clips):

Depth codeAbsRel after VAE, real / RLBench
Ours (power transform → path through the RGB cube, one fixed 0.05–10 m range)0.75% / 0.38%
Same transform on a grey ramp2.06% / 0.72%
Turbo colour map1.86% / 0.81%
Inverse depth on a grey ramp, range fitted per dataset0.94% / 0.43%
24-bit packing into RGB (exact before the VAE)61.7% / 31.0%

Our code is the most accurate, with a single fixed range for every source; the same transform on a grey ramp is 1.9–2.8× worse, and the code that is exact on paper collapses after the VAE. For role maps, the fixed palette keeps the instructed cube at IoU 0.98 after the round trip, against 0.66 for grey-level indices.

(ii) Shared generation aligns geometry learning with action learning (Table 18). Every template is denoised by the same transformer and head, so a geometry-generation step updates the weights the deployed RGB+action query uses. On 40 real-robot windows we measure the cosine between the deployment action-loss gradient and each geometry-objective gradient (shared blocks 2/14/26, reliability-corrected):

Weights / geometry objectivedepthnormalsroles
XGenAct (generation)+0.12+0.09+0.43
Private-head variant (generation)+0.08+0.10+0.40
Auxiliary-head variant (auxiliary regression)−0.00+0.02+0.01
Auxiliary-head weights, generation (control)+0.01+0.02+0.05
Initial checkpoint, generation (control)+0.00+0.01+0.11

In models trained to generate geometry in the shared space, geometry and action gradients point the same way, so to first order every geometry update also improves the action. Auxiliary regression of the identical labels produces gradients unrelated to the action objective, and the alignment is near zero at initialization for depth and normals and much weaker for roles (+0.11 vs. +0.43): it is learned through shared generation. This provides a mechanistic account of the 19/20 vs. 13/20 result.

The plans use this geometry (Table 19). Erasing the linear geometry subspace from the observation tokens shifts the generated plan 4.2 cm more than erasing a random subspace of equal rank (95% CI 2.2–6.8), and deleting the instructed cube shifts it 9.5 cm more than deleting background.

Relation to S-VAM and Cosmos Policy. S-VAM regresses frozen-foundation-model features of the backbone's own generated videos and feeds them to a separate action expert; no depth or semantic map is decoded. Cosmos Policy writes actions, proprioception, images and values into latent frames; it predicts no structured geometry. XGenAct makes metric depth, normals and functional roles decodable outputs of the same generator that produces the actions, with no modality-specific module (Appendix L; hLCN W3).

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all three reviewers for their careful and constructive reviews. Their central question is whether XGenAct's gains come from the shared generative formulation itself or merely from additional geometric and semantic labels. The revision answers it with a controlled experiment: with the labels, data and recipe held fixed, generating the geometry in the shared video space reaches 19/20 real-robot successes, against 13/20 when the same labels are used as conventional auxiliary targets and 9/20 with modality-specific output heads. XGenAct also outperforms four external policies trained on the same demonstrations (3/20–9/20), and new mechanistic analyses show why shared generation helps. All additions are in the revised PDF.

Videos of all 140 real-robot trials, for every method and including failures, are provided in the https://anonymous.4open.science/r/real-robot-eval-videos-0377 and supplementary material.

Three effects, now measured separately.

  1. Structured perception supervision vs. RGB-only training: Table 2, average success 32.5% → 52.5–61.25% under matched backbone, optimization and action budget.
  2. Generating the labels in the shared video space vs. regressing them as auxiliary targets: 19/20 vs. 13/20 (new).
  3. One shared output head vs. private heads: 19/20 vs. 9/20 (new).

Matched-supervision ablation (§4.4). Three models start from the same Action Images checkpoint and share data, labels, template menu, conditioning plans, optimizer, schedule, seed and action supervision. Only the way depth, normals and role maps enter training differs:

  • XGenAct (shared head): geometry is generated in the shared canvas through one output head.
  • Private-head variant: each stream type has its own copy of the pretrained output head (+0.037% parameters), starting from exactly XGenAct's function.
  • Auxiliary-head variant: geometry never enters the canvas; the same geometry latents are regressed by light auxiliary heads from the RGB tokens, on the same frames and with the same loss weight, and the heads are discarded at inference. This reproduces, inside our backbone, the auxiliary-target design of GeoSem-WAM, WAM4D and DreamWAM.

All three are deployed identically from the two current RGB frames and the end-effector pose; none receives depth, normals or segmentation at test time.

Result (Table 3). Real robot, instructed target colours never used as targets in training:

ModelGeometry used asSuccess95% CI (Wilson)
XGenAct (shared head)generated, one shared head19/2076–99%
Auxiliary-head variantregressed auxiliary target13/2043–82%
Private-head variantgenerated, per-stream heads9/2026–66%

Shared generation outperforms auxiliary regression of the identical labels (p = 0.04) and private output heads (p = 0.001; two-sided Fisher exact tests). The benefit therefore comes from how the labels are used, not from the labels themselves. Appendix M explains why: only shared generation aligns geometry learning with action learning (nfba W4).

Scope. The ablation uses one training seed per model and 20 trials per method; its RLBench counterpart will be added to the final version.

AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions

We thank the reviewers again for their constructive feedback, which substantially strengthened the paper. The table below lists all revisions and where they appear in the revised manuscript; each is discussed in the individual responses. Section, table and appendix numbers refer to the revised manuscript.

#RevisionLocation in the revised manuscriptAddresses
1Matched-supervision ablation§4.4, Table 3, App. Jnfba W2/Q2, jWPj W1/Q1, hLCN W1/W4
2Real-robot evaluation with four external baselines§4.4, Table 3, Fig. 4, App. Jnfba W3, hLCN W3
3Mechanism: codec ablation, gradient alignment, causal interventionsApp. M, Tables 17–19nfba W4, hLCN W1
4Functional-role label provenance; test-time privilege auditApp. ChLCN W2
5Action-codec and frozen-VAE round-trip fidelityTable 5nfba Q3
6Per-query latency, denoising-step trade-off, 4/6/10-stream costApp. K, Tables 15–16nfba Q3, jWPj W2/Q2
7Loss-share diagnostics and template-ratio sensitivityApp. DjWPj W3/Q3
8Unfiltered perception results on all four tasks (main-text figure) and on 14 held-out tasks§4.5, Fig. 5, App. I, Tables 9–10jWPj W4/W5/Q4
9Baseline protocol matrixTable 7nfba W1/Q1, hLCN W3
10Cross-prompt action consistencyApp. KjWPj W2
11Comparison with geometry-aware WAMsApp. Lnfba W4, hLCN W3
12Wording aligned with the evidence: Table 1 claim (Evo-1 removed), objective description, role-label provenance, cross-prompt consistencyAbstract, §1, §3.4, Table 1, App. C/Knfba W1, jWPj W2/W3, hLCN W2

Dear Reviewers,

Thank you again for your time and thoughtful feedback. With less than 12 hours remaining in the discussion period, we would sincerely appreciate knowing whether our response has addressed your main concerns.

Please also let us know if you have any remaining questions or if there are any points we could further clarify. We would be grateful for the opportunity to provide any additional information before the discussion concludes.

Thank you again for your careful consideration of our work.

Sincerely,
The Authors of Submission 19670

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.