acceptodds
Withdrawn from ICLR 2027

X-World: Real-Time Cross-Embodiment Policies with Scalable Video World Priors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-instructed manipulation requires robust closed-loop policies, yet action-labeled robot trajectories are costly and limited in coverage. Human and robot videos provide rich spatiotemporal interaction priors that action supervision alone cannot easily capture. Transferring video diffusion priors to policy learning must accommodate variable camera layouts, heterogeneous state and action interfaces, and real-time control. We propose X-World, which adapts a pretrained video diffusion backbone to variable-view robot observations, distills its hidden states into a compact token sequence with a Q-Former, and trains a flow-matching action expert that attends to this sequence to predict action chunks. Dataset-specific modules align heterogeneous state and action interfaces. We first adapt the video diffusion backbone on large-scale robot and human videos, freeze it while training the Q-Former and action expert with an action-only objective, and then post-train on the target platform. At inference, X-World maintains a 20Hz model response rate and uses asynchronous real-time chunking to decouple model updates from action execution, avoiding waiting latency. It shows strong performance on in-distribution LIBERO, zero-shot LIBERO-Plus distribution shifts, and real-world manipulation tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World action model, cross-embodiment, real-time

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_dQVdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes X-World, using a Q-Former to extract a compact world prior from a video generation backbone and condition an action DiT for continuous action prediction. The model is trained on heterogeneous robot and human videos and robot action datasets, achieving strong performance on LIBERO, LIBERO-Plus, and real-world manipulation tasks.

Critical strengths
  • The method retains pretrained self-attention while enabling cross-view fusion through an additional zero-initialized residual attention module.

  • Real-world rollouts demonstrate coordinated bimanual manipulation. The reported inference speed and asynchronous execution are practical strengths.

Critical weaknesses
  • The paper emphasizes cross-embodiment learning, but policy evaluation is limited to LIBERO/LIBERO-Plus and a dual-arm UR5e platform. Evaluation on more diverse embodiments would strengthen this claim.

  • For the variable-view design, disabling residual cross-view attention does not isolate the benefit of retaining pretrained self-attention. A matched comparison with the Genie-Envisioner-style replacement would better support this design choice.

  • For the real-world experiments, the evaluation conditions are matched, but baseline training details remain unclear. Please specify the checkpoints, demonstration data, and fine-tuning budgets used for each baseline.

  • Dataset-specific projection heads leave transfer to new action/state interfaces unclear. Although target-platform post-training is evaluated, an explicit held-out-embodiment study with varying demonstration budgets, as explored in X-VLA, would better establish adaptation capability.

  • The Q-Former ablation uses only the backbone's last-layer features, whereas the full model uses all layers (Sec. 4.4). This confounds compression with access to more features; a comparison using the same backbone layers would be more informative.

Questions for authors
  • Could the authors report variation across training seeds for the small LIBERO gains, particularly 98.5% versus 98.1% in Table 1 and 98.5% versus 98.3% for the video-loss ablation?
Reviewer_wYPzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

X-World studies language-conditioned manipulation using representations from a pretrained video diffusion model. It adapts LTX-Video to multiple camera views through residual cross-view attention, compresses features from its transformer layers using a Q-Former, and conditions a flow-matching action expert on these tokens. Dataset-specific input and output modules accommodate heterogeneous robot interfaces.

Training consists of video adaptation, action pretraining with the video backbone frozen, and joint target-domain post-training. Deployment uses asynchronous action chunking.

The paper reports 98.5% average success on LIBERO, compared with 98.1% for the strongest listed baseline; 76.5% on zero-shot LIBERO-Plus, compared with 69.6%; and 81.7% across six dual-arm UR5e tasks, compared with 58.3% for the strongest evaluated baseline. These results appear in Tables 1–2 and Figure 4.

Critical strengths

A practical interface between video pre-training and heterogeneous robot control. The method accommodates different camera configurations and robot action interfaces around a shared backbone. Its compressed video representation gives the action model a fixed interface despite changing visual inputs. This addresses concrete obstacles to using video models across robot datasets (section 3.2). The contribution is the integrated design; the individual components have precedents.

Validation beyond a nearly saturated simulation benchmark. The paper evaluates both zero-shot robustness under LIBERO-Plus perturbations and coordinated manipulation on real hardware. The substantial hardware advantage under shared evaluation conditions provides the strongest evidence that the complete system is useful. This matters more than its small lead on standard LIBERO (section 4.2–4.3).

Critical weaknesses

The ablations leave the main architectural and training contributions unresolved. Removing the Q-Former also replaces features from multiple video-backbone layers with features from only the final layer. This changes both the information available to the policy and how it is compressed. The comparison therefore cannot establish whether layer-aligned compression provides an advantage over a simpler method using the same features. The paper also emphasizes preserving video priors by freezing the backbone during action pretraining, but never compares this with joint training. These design choices are not separately evaluated on LIBERO-Plus or the main real-world tasks. Consequently, the results support the complete pipeline more strongly than the specific compression and training strategies presented as its contributions

Questions for authors

During Stages 2 and 3, are the features conditioning the action expert always extracted from independently sampled future noise at σ=1\sigma=1, matching inference? Or can they depend on partially noised ground-truth future videos? Please specify the computation graph, including whether video and action losses use separate backbone passes. A matched, observation-only conditioning path would resolve an important uncertainty.

For each real-world baseline, please provide the checkpoint, demonstrations used, trainable parameters, optimization settings, training duration, and checkpoint-selection procedure. Were all baselines adapted using the same demonstrations? These details materially affect interpretation of the 23.4 percentage-point advantage.

Can the Q-Former comparison retain the same multilayer inputs while changing only the compression mechanism? What evidence supports freezing during Stage 2 over joint training? If these comparisons are unavailable, narrowing the claims about layer alignment and “knowledge isolation” would make the conclusions better supported.

What value of E is used, what is the measured refresh frequency in the deployed loop, and how are committed prefixes clamped during action sampling? Please distinguish uninterrupted execution from the delay before a new observation can affect an executed action.

Reviewer_DJcMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts a pretrained LTX-Video backbone to variable-view robot observations with a zero-initialized residual cross-view attention branch and view embeddings, distills per-layer denoising hidden states into a fixed-length token sequence with a layer-aligned Q-Former, and trains a flow-matching action expert with dataset-specific state/action heads on top. Training is three-stage (video adaptation on robot and human video, frozen-backbone action pretraining, joint post-training), and deployment uses asynchronous real-time chunking. The method reaches 98.5% on LIBERO, 76.5% on LIBERO-Plus zero-shot, and 81.7% on six real-world dual-arm tasks, with a 46 ms model response time.

Critical strengths

The zero-init residual cross-view branch is a clean way to preserve the pretrained backbone at initialization, and the paper is one of the few in this line to report concrete deployment latency with a working asynchronous chunking scheme on real hardware.

Critical weaknesses

After reading the paper and the experiments, I am still not fully convinced why a specifically designed training algorithm for variable-view is even needed, given the tremendous training overhead of adapting pretrained video models into a multi-view version (Stage 1 + Stage 2 alone are 64 H100 x 10 days, before post-training).

I think the paper also lacks proper benchmarks (both environments and baselines). The paper does discuss prior WAM/VAM works but does not properly compare against them (e.g. Motus, mimic-video, LingBot-VA, Cosmos Policy), which makes the results hard to calibrate. For example, LingBot-VA (RSS 2026) and Cosmos Policy (ICLR 2026) both reach 98.5 on LIBERO ID, on par with this paper, and both utilize additional views with much simpler and cheaper recipes (expanding the video spatially or temporally, respectively, without additional architectual changes). Cosmos Policy in particular reaches the same LIBERO average from a single post-training stage of 64 H100 x 48 h. Both works are already cited in Related Work, so their absence from Tables 1 and 2 is hard to justify. The LIBERO-Plus comparison has the same problem: the baseline set in Table 6 is exactly the set evaluated in the LIBERO-Plus paper, and the paper's own strongest LIBERO competitors (X-VLA, pi0.5, Genie-Envisioner, all of which the authors already run on the real robot) are missing from it. The real-world experiments still have the same issue: modern WAM/VAM baselines are not included. Without these baselines, I can't assess whether the gain comes from simply utilizing the weights of video models or whether the multi-view-aware design choices were actually important.

The ablation studies also look suspicious because the most hard-to-believe result is not included in the ablation table but rather described in a sentence in Section 4.4: "Disabling the residual cross-view attention reduces average success to 83.1%, while removing the view embedding reduces it to 93.3% (vs. 98.5% for the full model). On Fold Towel, these variants succeed in 0/20 and 2/20 trials, respectively." Since the baselines that would defend this work's design choices are missing, I would suspect the loss from disabling cross-view attention to be more of a training difficulty of the proposed approach, rather than a justification that the multi-view design is actually important for learning WAMs/VAMs. A 2B video backbone with ten days of pretraining landing at 83.1 on two-view LIBERO is below fine-tuned Octo, and the related Table 3 entry where adding Stage 2 on top of raw LTX (89.5) is worse than skipping both stages (92.1) points in the same direction.

Questions for authors
  1. If a frame-level or width-concatenation multi-view variant of LTX (the Cosmos / LingBot recipe) can be trained under the same three stages, that single baseline would settle the main question of this review. Could the authors add this baseline?
  2. For the w/o cross-view and w/o view-embedding variants: were training budgets and hyperparameters matched to the full model, and how do the two views enter the Q-Former when the residual branch is disabled? If applicable, I think it would be bette to move these numbers into Table 3 with per-suite numbers.
  3. Why does Table 2 omit the Camera and Noise shifts, on which X-World is below OpenVLA-OFT and pi0-Fast? For a method whose first contribution is view handling, the camera-shift weakness deserves discussion.
  4. Were the 20 real-world initial placements present in the 30-minute teleoperation data?
Reviewer_RCH1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

X-World transfers a pretrained video diffusion model to robot control. The method consists of three main components: a variable-view video diffusion backbone using zero-initialized residual cross-view attention and view embeddings; a layer-aligned Q-Former that compresses intermediate video representations into compact tokens; and an action expert with dataset-specific state and action projections. Training has three stages: video adaptation, action pretraining with the backbone frozen, and joint target-domain post-training. Deployment uses one backbone pass and asynchronous action chunking.

Experiments demonstrate competitive performance on LIBERO and LIBERO-Plus, and strong real-world performance across six dual-arm manipulation tasks. Ablation studies examine the contributions of video pretraining, cross-view attention, the Q-Former, dataset-specific projections, and real-time chunking.

Critical strengths
  1. Large real-world lead. X-World achieves 81.7% average success, compared with 58.3% for X-VLA, the strongest reported baseline. Furthermore, the system integrates asynchronous action chunking with a measured 46 ms model response and continuous 30 Hz action execution.

  2. Effective multi-view adaptation. The zero-initialized residual cross-view attention provides a practical way to fuse multiple camera views while retaining pretrained self-attention. Removing cross-view attention reduces LIBERO success from 98.5% to 83.1%, and the variable-view evaluations show that the model remains functional when camera views are missing. These results support the importance of cross-view fusion for robotic manipulation.

Critical weaknesses
  1. Insufficient isolation of architectural contributions. Genie-Envisioner already employs selective cross-view attention, a frozen video backbone during action pretraining, and layer-wise video-to-action conditioning. X-World's residual cross-view design is ablated only against removing cross-view fusion, leaving its specific advantage over alternative designs unclear. Furthermore, since Stage 1 trains both the video backbone and the newly introduced cross-view module, its 9-point improvement does not isolate the benefits of temporal video-prior learning from those of cross-view adaptation.

  2. Limited evidence for heterogeneous action pretraining. Removing Stage 2 or replacing dataset-specific projections with shared ones each reduces LIBERO success by only 1.1 points on a nearly saturated benchmark. Interestingly, removing both Stage 1 and Stage 2 (92.1%) outperforms removing Stage 1 alone (89.5%), an unexplained interaction. Without corresponding ablations on LIBERO-Plus or the main real-world tasks, the contribution of heterogeneous action pretraining to the larger performance gains remains unclear.

Questions for authors
  1. Table 3 shows that removing both Stage 1 and Stage 2 achieves higher LIBERO success (92.1%) than removing Stage 1 alone (89.5%). What explains this result? Does it suggest that heterogeneous action pretraining can cause negative transfer without prior video adaptation?

  2. In the "w/o Stage 1" ablation, is the zero-initialized cross-view residual branch frozen and thus inactive throughout Stage 2, becoming trainable only during Stage 3? If so, does this ablation conflate the absence of video-prior adaptation with the absence of pretrained cross-view fusion?

  3. Do the view embeddings represent physical camera identities or input slot indices, and how are they assigned in the shuffled-view evaluation? Could the authors clarify how this design explains the unchanged shuffled-view performance alongside the substantial degradation when view embeddings are removed (LIBERO: 98.5% to 93.3%; Fold Towel: 12/20 to 2/20)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.