acceptodds
Under review as a conference paper at ICLR 2027

Xperience-0: Scaling Loco-Manipulation World Models across Embodiments with Human Experience

16 views

Abstract

World models enable embodied agents to anticipate action consequences, yet jointly modeling locomotion and manipulation across diverse embodiments remains challenging due to heterogeneous action spaces and limited multimodal supervision. We present Xperience-0, the first egocentric loco-manipulation world foundation model spanning humans and robots with diverse embodiments. We curate Xp-Data, comprising 10k hours of egocentric human data and 3k hours of robot data with unified multimodal annotations and systematic quality control. A unified kinematic representation encodes locomotion through camera poses and manipulation through projected skeletons, with robot skeletons derived from embodiment-specific forward kinematics. We also introduce Xp-Teleop, a long-horizon loco-manipulation benchmark based on high-quality dexterous humanoid teleoperation data. Evaluations demonstrate state-of-the-art performance on the DreamDojo, , -HOME, and Xp-Teleop benchmarks, from zero-shot inference to few-shot and full-data post-training. Scaling human pre-training data improves performance on both human and robot evaluations. After distillation, our model supports real-time, minute-long rollouts, enabling live VR interaction. Applications in policy evaluation, model-based planning, and world–action modeling further demonstrate the model's utility for embodied decision-making.

Primary area:
generative models
Keywords:
world models, video generation, cross-embodiment learning, loco-manipulation, embodied intelligence, robot learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dCJnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an ego-centric loco-manipulation world foundation model with embodiments spanning across humans (10k hours) and robots (3k hours). An unified kinematic representation encodes locomotion through camera poses and manipulation through projected skeletons. It also introduces a long-horizon loco-manipulation benchmark based on dexterous humanoid teleoperation data. Evaluations show state-of-the-art performance on several benchmarks, ranging from zero-shot inference to few-shot and full-data post-training.

Critical strengths

This paper identifies a clear gap in current experimental robotics: lack of high quality curated datasets and controlled benchmarks, spanning multiple embodiments and domains, and proposes a complete pipeline to address this gap. It contains human-in-the-loop for quality checks, temporally aligned camera trajectories, captions, long-horizon dexterous teleoperation with locomotion and manipulation, etc. The contributions are straightforward and the paper is clear about that. Ablations are well positioned and clearly show the benefits of each design choice.

Critical weaknesses

The main weakness of the paper, in my opinion, is the lack of theoretical contributions, given that the targeted venue is ICLR. As I said above, the paper is clear about its contributions and they are relevant to the community, however I don't think there is enough novelty in each component individually: skeleton projection is well-known, plucker-ray maps are standard, so are spatially-aligned conditioning, human pretraining + human-robot (or robot) mid-training is a common recipe, and DMD for real-time interactive rollouts are common place at this point. Improvements in different benchmarks is expected given enough engineering effort and high quality datasets, which again, is worthy contribution in an of itself, but does not bring theoretical insights.

Questions for authors

Broadly speaking, I would like to hear from the authors what are the insights one could gain from this study, that could guide the development of new solutions to these problems (in particular the downstream applications mentioned in Section 5).

3 comments
AuthorsOpenReview ↗

Response to Reviewer dCJn (1/2)

We sincerely thank Reviewer dCJn for recognizing our data pipeline, quality control, and benchmark as meaningful contributions, and the ablations as well positioned and clearly showing the benefit of each design choice. Added content in the revised paper is marked in blue.

W & Q: The main weakness is the lack of theoretical contributions; there is not enough novelty in each component individually. What insights from this study could guide new solutions, in particular for the downstream applications in Section 5?

A: Our contribution is an integrated framework: a unified kinematic interface that, to our knowledge, is the first to let one world model predict egocentric loco-manipulation across humans and robots, together with the data, benchmark, and insights that make this possible. The framework is not a simple assembly of existing techniques: its key choices are made against alternatives, the gaps of the chosen representation are identified and addressed, and substantial data work makes the geometry correct. Below we give the positioning, the central insight with its evidence, the design principles that follow, the limits of image-space interfaces and our extensions, and the insights for downstream use.

1. Positioning.

  • Full setting. A cross-embodiment world model for egocentric loco-manipulation: locomotion moves the camera while manipulation moves the body in view, and the two are predicted jointly; one model serves humans, grippers, and dexterous hands, each driven by its own native kinematics.
  • Closest works cover only parts of this setting. OSCAR targets tabletop manipulation with fixed-base arms and does not address locomotion (it filters for static cameras and has no camera-trajectory condition). AnyWorld also conditions on camera and hand skeletons, but its only human data is EgoDex, so the trained model remains focused on tabletop manipulation rather than loco-manipulation; and it drives dexterous hands from a source human-hand skeleton, so the model is in effect a human-to-robot retargeter, whereas ours is driven by the target robot's own actions across grippers and three- and five-finger dexterous hands. DreamDojo and Cosmos3 condition on latent actions or robot-specific action vectors rather than a shared geometric interface, so each new embodiment requires a dedicated control interface or additional adaptation.

2. Central insight: a simple geometric interface is sufficient for cross-embodiment world modeling.

  • Decoupled representation. Loco-manipulation decomposes into its two motion sources, and each gets its own geometric channel: locomotion is expressed as metric camera motion, identical in form for every embodiment, and manipulation as each body's native projected skeleton, embodiment-specific in content but shared in format. The decoupling is at the input only: the model still predicts their joint effect, with locomotion-induced viewpoint changes and manipulation unfolding together in one video.
  • No action mapping to learn. With actions expressed in the observation's own geometry, no embodiment-specific action mapping needs to be learned: constructing the conditioning inputs for a new body requires only its URDF forward kinematics and camera calibration—an analytic-geometry problem, not a learning problem; what target data must still teach is the embodiment's appearance and visual dynamics.
  • Metric scale is key. Unlike many camera-controlled video models trained on scale-normalized relative poses, our camera translations are in meters, consistent with the FK skeleton, robot commands, and VR headset poses, so conditions can be computed directly from commands (Table S8).
  • Direct evidence. On two robots held out from all our training data, our human-video-only 5B model follows their camera motion at 0.016 m ATE, ahead of Cosmos3 fine-tuned on robot data (0.025–0.031 m) and pre-trained DreamDojo (0.033 m) (Table S6). One epoch of target data then lifts appearance from 19.73 to 22.38 dB PSNR and tightens ATE to 0.008 m. The skeleton carries the rest of the action: removing it costs 3.8/1.8 dB for A14B/5B (Table S15).
  • Correct geometry is the precondition—hence the data contribution. Camera trajectories must be metric and skeletons must align with the hand. On the camera side, we annotate metric trajectories for every source (from available calibration or our SLAM pipeline), with quality control and unified units and coordinate frames. On the skeleton side, many robot datasets ship without the calibration (e.g., hand–eye extrinsics) needed for FK skeletons to project accurately onto the released frames (Table S2); we calibrate the required transforms for our own collections and manually annotate or correct image-space keypoints where accurate projection is unavailable (Sec. 3; App. A).
AuthorsOpenReview ↗

Response to Reviewer dCJn (2/2)

3. Design principle: unlocking foundation-model capabilities through a simple interface.

  • Simple, architecture-agnostic interface. We deliberately avoid specialized modules: a simple, effective interface unlocks the capabilities of already-strong video foundation models, so stronger base models translate directly into better world models. Scaling from 5B to A14B improves prediction in every regime: +2.2 dB on the cross-embodiment set (Table S15), +2.3 dB in-distribution on AgiBot (Table S14), and, without post-training, +2.1/1.7/1.4 dB on Ψ0\Psi_0/ω\omega-HOME/Xp-Teleop (Table 3 of the paper).
  • Skeletons over mesh renderings. We weighed richer renderings such as URDF mesh masks and chose skeletons: a mesh also carries shape and contour—useful for a single robot but embodiment-specific—whereas a skeleton keeps only the kinematic structure, which human hands and robot FK keypoints share in form; human video can therefore pre-train the same interface directly, and human-video-only pre-training already follows unseen robots' camera motion zero-shot (Table S6; arm following in Figs. S16–S17).
  • Injection follows spatial alignment. We also compared how each condition should enter the backbone: pixel-aligned camera (Plücker), skeleton, and wrist-pose maps are fused directly with the latents—addition and concatenation perform comparably (Table S19), as expected since a linear layer on concatenated inputs decomposes into a sum of per-input projections—whereas the wide-FOV skeleton extends beyond the frame and is not pixel-aligned, so it enters via cross-attention.

4. Limits of image-space interfaces, and how we extend them. A projected skeleton describes only what lies inside the image; we address three gaps that prior work leaves undiscussed:

  • Depth ambiguity. For robots the skeleton comes from forward kinematics, so it stays accurate even under occlusion; what cannot be known in advance is the relative depth between the manipulator and the scene, and the model learns to resolve this ambiguity from large-scale human and robot data.
  • Out-of-view context. An arm outside the frame still determines what enters the view, but an in-frame skeleton cannot represent it; since the FK skeleton remains accurate beyond the field of view, we render it with a wider field of view (1.5×). On RoboTwin 2.0, where the in-frame skeleton touches the image border in 99.4% of frames, this raises manipulator coverage from 0.37 to 0.56 and recovers the off-frame arm (Figs. S35–S36).
  • Orientation ambiguity. A projected skeleton leaves the wrist's 3D orientation ambiguous; wrist-pose renderings supply the missing orientation explicitly.

5. Insights for downstream use.

  • One evaluator across embodiments. Since conditions depend only on URDF and calibration and can be computed from commands (Table S8), the same interface and architecture evaluate π0.5\pi_{0.5} on an arm and Ψ0\Psi_0 on a humanoid, with no per-robot action head or encoder.
  • Relative rather than absolute judgments. Predicted and real outcomes are highly correlated in all three settings—task progress on RoboTwin (r = 0.984) and on the real G1 (r = 0.996 with the teleoperation reference, 0.991 without), and success rates on SIMPLE (r = 0.963)—with identical checkpoint rankings (MMRV = 0) even where absolute values differ (Fig. 4, Table S7, App. K); the model is thus best used for checkpoint selection and candidate comparison.
  • Planning over multiple policies. Value-model selection over world-model rollouts consistently outperforms the best fixed checkpoint (92 vs. 85% task progress on RoboTwin, 55 vs. 45% on the real G1, and 80 vs. 70% success on SIMPLE; Fig. 4(c–d), Fig. S19), suggesting the world model exploits the complementary strengths of different policies.
  • Priors pay off in generalization. As FastWAM's backbone, our model barely changes in-distribution LIBERO (96.95 → 97.00%) but improves LIBERO-Plus (50.06 → 56.82%), every few-shot setting (Table S13), and real-G1 task progress (33.3 → 56.7%, Fig. S20). World models should thus be judged by downstream utility, not video fidelity alone.
  • Live loco-manipulation with humans as an embodiment. Prior live teleoperation of world models (e.g., DreamDojo) controls only the robot's upper body; in Xperience-0, the VR user's head motion drives the egocentric camera and hand motion drives the skeleton, so one model lets the user act as a human or a robot in real time (21.3 fps) over minute-long rollouts (Fig. S10).

We have consolidated these points in a new Findings section (App. R) of the revised manuscript, and we are committed to supporting the community through open-sourcing. We would be glad to discuss any of these points further.

AuthorsOpenReview ↗

Follow-up on rebuttal

Dear Reviewer dCJn,

With the discussion deadline approaching, we would appreciate your feedback on our rebuttal and revised manuscript at your earliest convenience. We have provided additional evidence and clarifications in response to your comments, and would particularly welcome an indication of which concerns, if any, remain unresolved, so that we have time to address them before the discussion closes.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_E7rcOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents Xperience-0, a set of egocentric local-manipulation world models, trained on 10K hours of human data and 3K hours of robot data from different embodiments. In addition to self-collected data, they post-processed multiple public datasets to unify geometric, kinematic, and semantic annotations. Xperience-0 is a video diffusion model conditioned on an initial image and text prompt. Trajectories of camera pose, a skeleton representation of hand pose, along with other variables are also conditioning variables for the model. The model is pretrained on human data, and midtrained on a mix of human and robot data. The model is then distilled into a student model that enables fast inference.

Xperience-0 demonstrated strong performance on a number of benchmarks. Xperience-0 is also evaluated on a number of downstream applicants, including policy evaluation, test-time planning, and etc. Notably, Xperience-0 shows signs of cross-embodiment generalization and serves as a strong backbone for world-action model.

Critical strengths
  1. The paper presents the first egocentric loco-manipulation world model. The method and experiments are well designed.

  2. The work is backed by extensive empirical evidence.

Critical weaknesses

None.

Questions for authors

Suggestions:

  1. It should be clear over what prediction horizon PSNR and other world model metrics are calculated over.

Clarifications: 2. Is the kinematic skeleton representation of hands morphology-agnostic? 3. Why not predict proprioception with the world model? 4. Eq (4) - Is the plus sign meant to be addition or concatenation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer E7rc

We sincerely thank Reviewer E7rc for the positive assessment—recognizing Xperience-0 as the first egocentric loco-manipulation world model with a well-designed method and experiments—and for the helpful suggestions. We address all questions below; added content in the revised paper is marked in blue.

Q1: It should be clear over what prediction horizon PSNR and other world-model metrics are calculated.

A1: Every single-window metric covers the whole generated window, with the same frames for every model: Human-200 uses 81-frame windows at 16 fps, scored on all 80 frames after the conditioning frame (≈5 s); the DreamDojo sets follow DreamDojo's native protocol (49 frames at 10 fps, ≈4.8 s, scored on all 49 frames); Ψ0\Psi_0/Xp-Teleop/ω\omega-HOME use 81-frame windows at 30 fps (≈2.7 s, scored on frames 2, 4, ..., 80); and the long-horizon results chain windows to ≈30–32 s. We state these horizons in App. L of the revised paper.

Q2: Is the kinematic skeleton representation of hands morphology-agnostic?

A2: It is morphology-agnostic in its interface and morphology-preserving in its content. Every manipulator is rendered with its native keypoints and connectivity into the same image-space skeleton channel: human hands (MANO keypoints), dexterous five- and three-finger robot hands (URDF forward kinematics), and parallel grippers. This channel passes through a single shared encoder and projection (PskelP_{skel} in Eq. 4), with no embodiment-specific parameters, encoders, or heads, so adding a new embodiment requires no architectural change. We deliberately do not retarget robot hands to a canonical human hand: the model renders each embodiment as it appears, including finger configuration and gripper aperture, which a canonical skeleton would discard. Xp-Data mixes five-finger hands, three-finger hands, and grippers through this one channel (Fig. S6 of the revised paper), and the same channel is used unchanged for two never-seen embodiments in the held-out-embodiment evaluation (Table S6).

Q3: Why not predict proprioception with the world model?

A3: In Xperience-0, embodiment kinematics are the input rather than the output: robot skeletons are rendered from joint states via forward kinematics (Eq. 3), and locomotion enters through camera poses; the conditioning signals thus encode the geometric and kinematic aspects of proprioception relevant to visual prediction, rather than the full proprioceptive state (e.g., torques, velocities, or contact forces). In policy evaluation and planning the action sequence is given and already specifies these kinematic signals; what remains uncertain is how the scene responds (object motion and contact outcomes), which is observable in pixels. Proprioceptive spaces are also embodiment-specific (different joint counts and layouts across G1, GR-1, and gripper arms) and the human videos carry no proprioception, so predicting it would require per-embodiment output heads that the shared interface is designed to avoid. The backbone is nevertheless compatible with predicting states or actions: initializing the world–action model FastWAM from our 5B backbone raises LIBERO-Plus success from 50.06% to 56.82% (Sec. 5.3, Table S13 of the revised paper).

Q4: Eq. (4)—is the plus sign meant to be addition or concatenation?

A4: It is addition: each condition latent has its own linear projection, and the projected tokens are summed with the video tokens. We also compare additive fusion with concatenation fusion, two variants trained with identical data and optimization and differing only in the fusion operator (Table S19 of the revised paper). The two operators perform comparably: additive is slightly better on Human-200, Ψ0\Psi_0, and Xp-Teleop, while concatenation is slightly better in PSNR on ω\omega-HOME. This comparability is expected: a linear layer applied to concatenated inputs decomposes into a sum of per-input projections (W[x;c]=Wxx+WccW[x;c] = W_x x + W_c c), so the two operators realize the same function class at the injection point. We adopt addition as it is simpler: it adds no input channels to the backbone and allows zero-initialized projections. We have stated this explicitly under Eq. 4 in the revised manuscript.

AuthorsOpenReview ↗

Follow-up on rebuttal

Dear Reviewer E7rc,

With the discussion deadline approaching, we would appreciate your feedback on our rebuttal and revised manuscript at your earliest convenience. We have provided clarifications in response to your questions, and would welcome any further feedback on whether our responses address them.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_Hq7DOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper presents Xperience-0, an egocentric video world model for cross-embodiment locomotion and manipulation. It uses a shared conditioning interface comprising camera trajectories and projected manipulator skeletons, and adapts pretrained Wan2.2 video backbones through human-video pretraining, human–robot mid-training, and optional target-specific post-training. The authors curate Xp-Data, containing approximately 10,000 hours of human and 3,000 hours of robot video, and introduce the Xp-Teleop benchmark. Evaluation covers motion-conditioned video prediction, cross-dataset transfer, long-horizon rollouts, few-step distillation, and downstream planning and policy-learning applications.

Critical strengths
  1. A practical shared interface supported by substantial data curation. Expressing locomotion through camera trajectories and manipulation through projected native skeletons provides a consistent input format across humans, grippers, and dexterous robots. The unified annotation pipeline and Xp-Teleop collection are useful contributions, particularly for studying simultaneous viewpoint changes and manipulation.

  2. Broad empirical evidence of improved prediction and adaptation. The paper evaluates multiple datasets and adaptation budgets, with particularly strong results for the A14B model. Importantly, it includes a same-architecture public-backbone comparison, a Cosmos3 comparison using the same robot mixture, and a 1,000-episode WorldArena evaluation. The FastWAM initialization results also provide evidence of downstream utility: full-data LIBERO-Plus success improves from 50.06% to 56.82%.

Critical weaknesses
  1. The independent contribution of the proposed training recipe and architecture is insufficiently isolated. Human-video pretraining, rendered kinematic conditioning, and camera-conditioned generation have precedents discussed in the paper. The architecture largely retains Wan2.2 and adds projected conditions and cross-attention. The existing ablations establish that motion conditions and domain pretraining are useful, but do not clearly establish the advantage of the proposed staging or fusion design over alternatives under matched data and compute. More importantly, Appendix G measures scaling using successive checkpoints from training as additional video hours are processed. This confounds data exposure with optimization compute, so the curves do not isolate the effect of increasing the amount of distinct human data. This matters because scaling human experience is central to the paper’s contribution.

  2. The evidence for deployable cross-embodiment prediction and decision-making remains incomplete. The world model conditions on future camera trajectories and skeleton sequences, which can be derived from recorded motion in offline evaluation. For planning, however, these conditions must be available before executing a candidate, and the paper does not sufficiently explain how future camera motion is obtained from commands or how command–execution discrepancies are handled. Moreover, “zero-shot” denotes an unseen target dataset while the target embodiment may already occur in mid-training; this supports cross-dataset transfer but provides limited evidence for generalization to entirely unseen embodiments. The downstream real-robot results are promising, but aggregate task-progress scores and correlations across only a few checkpoints, without sufficient trial counts, uncertainty estimates, or planning details, leave the decision-making claims difficult to assess.

Questions for authors
  1. Clarification of the scaling evidence: Are the scaling curves exclusively successive checkpoints from one training run? Do you have comparisons that vary the amount of distinct human data while holding optimization steps and compute fixed? Please distinguish improvements from additional training from improvements attributable to broader data coverage.

  2. Clarification of planning inputs: For each candidate plan, how are the future camera trajectories and skeletons constructed before execution? Which signals are commanded targets versus recorded realized states? Please also clarify whether planning prompts contain only prior task instructions or information derived from the eventual execution.

  3. Evidence for the training and transfer claims: Do you have compute-matched comparisons between staged training and joint human–robot training from the outset, or evaluations excluding all training data from a target embodiment? These results would help determine whether the gains establish the proposed training recipe and unseen-embodiment generalization.

  4. Clarification of downstream evaluation: Please provide the real-robot task definitions, trial counts, task-progress rubric, uncertainty estimates, candidate-generation procedure, planning horizon, and replanning frequency. Does the reported real-robot Pearson correlation include the teleoperation reference point shown in Fig. 4?

10 comments
AuthorsOpenReview ↗

Response to Reviewer Hq7D (1/7)

We sincerely thank Reviewer Hq7D for the careful and constructive review, and for recognizing the practical shared interface, the data curation behind it, and the broad empirical evidence. We first answer the four questions; the new experiments there also ground our responses to the two weaknesses (W1–W2), which we summarize at the end. Added content in the revised paper is marked in blue.

Q1: Are the scaling curves successive checkpoints of one run, and can the effect of distinct human data be separated from additional training at fixed compute?

A1: Yes to both: the curves are successive checkpoints of a single pre-training run, and a new compute-matched control isolates the effect of distinct data at equal total optimization.

  1. Original curves. The scaling curves (Figs. S12–S14, App. G) are successive checkpoints from a single pre-training run with a fixed data mixture. The x-axis reports distinct human data; no clip is repeated within the first epoch, so there it coincides with the hours processed. We agree, however, that optimization compute grows along the same run, so these curves alone do not fully separate data coverage from additional training.
  2. Compute-matched comparison. We take the snapshots underlying the scaling curves and continue each one, using only the data it has already seen, until every setting reaches the same total number of optimization steps as the final checkpoint; the final checkpoint (which has seen the full corpus) is the shared right-most point and, being already at that total, needs no continuation. The settings then differ only in the amount of distinct human data at equal total optimization. Each setting is evaluated at the weights reached exactly at this matched budget—no intermediate checkpoint selection is performed. Fig. S15 (App. G) overlays these compute-matched checkpoints (blue) on the original curves (orange).
  3. Results. At the same total optimization, prediction quality still improves as distinct human data increases: from the smallest to the full-data setting, every reported metric improves, with PSNR gains of +0.5 dB on held-out human video, +1.6–2.3 dB on DreamDojo GR-1, and +1.0–1.3 dB on Ψ0\Psi_0/Xp-Teleop/ω\omega-HOME. Meanwhile, at matched data amounts the extra optimization changes PSNR by at most +0.35 dB (mostly below 0.2 dB). The gains in Figs. S12–S14 are therefore attributable primarily to broader data coverage rather than to additional training. We have added this control to App. G (Fig. S15).
AuthorsOpenReview ↗

Response to Reviewer Hq7D (2/7)

Q2: How are future camera trajectories and skeletons constructed before execution, which signals are commanded vs. realized, and what do planning prompts contain?

A2: In short, all planning-time conditions are computed only from each candidate's commanded targets; no information from the eventual execution is used.

  1. Construction of camera trajectories and skeletons. Each candidate plan is an action chunk of commanded targets proposed by a policy from the current observation, before any execution: a base-velocity command (vx,vy,ωyaw)(v_x, v_y, \omega_{yaw}), waist/arm joint targets, and hand joint targets. The commanded base velocity is integrated to a base pose, which is propagated through the fixed kinematic chain (base → torso → head camera) using the commanded waist joints; the resulting camera poses are expressed relative to the first frame and encoded as Plücker-ray maps (Eq. 2). The commanded joint targets pass through forward kinematics and are projected with that camera following Eq. 3, using commanded rather than measured joints. This construction is now stated in App. K of the revised paper.
  2. Commanded vs. realized signals. In planning, candidates are generated before execution, so no information from the eventual execution exists at planning time. Policy evaluation follows DreamDojo: each checkpoint is deployed on the real robot, and the world model re-simulates the episode from the same first frame, with conditions rendered from the commands the policy issued during that rollout (commands only, never measured states). This tests whether the world model predicts the outcome of each command sequence faithfully enough to rank policies. The fully counterfactual setting, where nothing from the execution is available, is tested by planning (Fig. 4(d)): selection over not-yet-executed candidates still beats the best fixed checkpoint (55% vs. 45% task progress).
  3. Planning prompts. Prompts contain only the task instruction available before execution (e.g., "move forward to pick up the cracker box"). No captions or other information derived from the eventual execution are used.
  4. Effect of the command–execution gap. We use SIMPLE, a simulated humanoid loco-manipulation benchmark on the Unitree G1 that exposes both commanded and executed states. On 7 tasks (99 windows) we generate the same windows from the same first frame, changing only whether the conditions are realized (an oracle reference) or command-derived, and measure all metrics against the executed motion. Condition following is nearly identical; the 95% CI of every paired difference includes zero (table below; also Table S8 of the revised paper; qualitative example in Fig. S18, App. K).
  5. Downstream validation. Using only command-derived conditions, world-model policy evaluation preserves the checkpoint ranking (Pearson r = 0.963 command-derived vs. 0.967 realized, MMRV = 0), the episode-level judgments under the two condition types agree with each other on 46/50 episodes (92%, Cohen's κ\kappa = 0.83; Fig. S19(c)), and value-model candidate selection reaches 80% success vs. 70% (best fixed checkpoint) / 36% (random). Details and figures are in App. K (Fig. S19). (These five Ψ0\Psi_0 checkpoints are a different set from the three real-robot checkpoints in Q4.)
MetricRealizedCommand-derivedPaired diff. [95% CI]
Camera ATE (m) ↓0.0660.063-0.003 [-0.012, +0.006]
Camera RPE@1s (m) ↓0.1240.123-0.001 [-0.012, +0.010]
Camera RRE (°) ↓3.193.27+0.08 [-0.41, +0.49]
Hand [email protected] ↑0.7070.712+0.005 [-0.021, +0.030]
Hand detection rate ↑0.9590.953-0.006 [-0.022, +0.010]
Hand L2 (px) ↓41.441.9+0.5 [-2.6, +4.0]
AuthorsOpenReview ↗

Response to Reviewer Hq7D (3/7)

Q3: Are there compute-matched staged-vs-joint comparisons, or evaluations excluding all training data from a target embodiment?

A3: We address the two parts separately.

1. Staged vs. joint training. The staged schedule is a practical design choice, not a claimed contribution. To compare it against joint human–robot training from the outset, we train two models from the same Wan2.2-TI2V-5B initialization on the identical set of human and robot clips with identical optimization, differing only in data order (staged: human-only clips first, then a human/robot mixture; joint: the same clips globally shuffled). Both runs use a reduced but identical budget (matched compute), so absolute numbers are below those of our full models. The two recipes perform comparably (table below; bold marks the better value in each row, both if tied; also Table S5 of the revised paper). We adopt staging because the human pre-training stage needs to be run only once and can be reused across robot mixtures and target embodiments, and because it lets us study the effect of scaling human experience independently. We have revised Sec. 4.3 to present staging as a practical design choice rather than a contribution, and added this comparison (Table S5, App. H).

BenchmarkMetricStagedJoint
Human-200PSNR ↑16.2116.19
SSIM ↑0.5480.546
LPIPS ↓0.4060.399
Camera ATE (m) ↓0.03660.0347
Camera RPE (mm) ↓2.812.70
Camera RRE (°) ↓6.026.06
Hand [email protected] ↑0.6930.695
Ψ0\Psi_0 1-shotPSNR ↑13.3913.24
SSIM ↑0.5470.531
LPIPS ↓0.4820.502
Xp-Teleop few-shotPSNR ↑12.1812.11
SSIM ↑0.4060.400
LPIPS ↓0.4640.464
ω\omega-HOME few-shotPSNR ↑14.4814.66
SSIM ↑0.6380.639
LPIPS ↓0.4440.436

2. Generalization to unseen embodiments.

  • Already in the paper. For the human-only pretrain stage every robot is an unseen embodiment, and its zero-shot rows on Ψ0\Psi_0/ω\omega-HOME/Xp-Teleop (Table S9) outperform every baseline on all metrics.
  • New strict held-out evaluation. We evaluate two embodiments absent from all pre-training and mid-training data: Galaxea R1-Lite (90 windows) from the Galaxea Open-World Dataset and Marvin Tianji (30 windows), collected by ourselves; both are also excluded from the robot mixture used to fine-tune the Cosmos3 baselines. The table in Part 4/7 pools the 120 windows; bold marks the best within each block (also Table S6 of the revised paper). Without any training on these embodiments, our 5B model pre-trained only on human video follows the new robots' camera better than Cosmos3 models fine-tuned on robot data (0.016 vs. 0.025–0.031 m ATE); after one epoch of post-training, our A14B model is the best on all six metrics (22.38 vs. 20.18 dB PSNR). Qualitative results are shown in Figs. S16–S17; the evaluation is added as Table 4 (main text), Table S6, Figs. S16–S17, App. I.

To remove the ambiguity the reviewer notes, the revised paper now explicitly distinguishes an unseen dataset (a dataset never trained on, though the embodiment may appear through other datasets) from an unseen embodiment (all data from the target embodiment excluded from training).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses

We sincerely thank all reviewers for their efforts and constructive feedback. We are encouraged by the reviewers' recognition of the practical shared interface, the broad empirical evidence, the carefully curated data pipeline and benchmark, and the well-positioned ablations. Their suggestions have helped further strengthen the paper.

Updates Addressing Reviewer Hq7D

  • Compute-matched human-data scaling. We take the snapshots at increasing amounts of human video and continue each on only the data it has already seen, until every setting reaches the same total training steps as the final checkpoint. At equal total optimization, performance still improves as distinct human data increases—every reported metric improves from the smallest to the full-data setting—while extra optimization on unchanged data brings only limited gains. Results added in Fig. S15 (App. G of the revised manuscript).
  • Command-derived conditions for planning and evaluation. All planning and evaluation conditions are computed from commanded targets only. On SIMPLE (Unitree G1 loco-manipulation, 7 tasks, 99 windows), condition following with command-derived inputs matches the realized-condition oracle, with all paired confidence intervals including zero. Results added in Table S8 and Fig. S18; the construction of command-derived conditions is detailed in App. K.
  • Downstream validation on SIMPLE. Policy evaluation correlates with simulator outcomes (r = 0.963), and value-model selection reaches 80% success vs. 70% for the best fixed checkpoint and 36% for random selection. Results added in Fig. S19.
  • Staged vs. joint training. Under matched data and compute, the two recipes perform comparably; staging is not claimed as a contribution. Results added in Table S5.
  • Generalization to unseen embodiments. New evaluation on Galaxea R1-Lite and Marvin Tianji, with both embodiments excluded from all training; zero-shot, our A14B model is already the best on all metrics. Results added in Table 4 of the main text, with the full comparison in Table S6, Fig. S16, and Fig. S17.
  • Real-robot and simulation downstream protocols. Added full protocols for real-robot policy evaluation (with confidence intervals), real-robot planning, RoboTwin, and SIMPLE, and corrected the description of the Pearson correlation in Fig. 4(b). Results added in Table S7; clarified in Sec. 5.3.
  • Fusion design. Additive and concatenation fusion perform comparably under matched budgets—as expected, since a linear layer on concatenated inputs decomposes into a sum of per-input projections. Injection follows spatial alignment: pixel-aligned conditions are fused directly, and the wide-FOV skeleton enters via cross-attention. Results added in Table S19.

Updates Addressing Reviewer E7rc

  • Clarified the prediction horizon and frame rate of each benchmark, including long-horizon (≈30 s) rollouts.
  • Clarified that the interface is morphology-agnostic while preserving each embodiment's native kinematic structure (no retargeting).
  • Clarified the role of kinematics vs. proprioception, supported by the FastWAM results.
  • Fusion comparison provided in Table S19.

Updates Addressing Reviewer dCJn

  • Clarified the central insight: with metric camera motion and native projected skeletons, constructing the conditioning inputs for a new embodiment becomes an analytic-geometry problem rather than a learning problem, supported by human-video-only pre-training that follows unseen robots' camera motion zero-shot (Table S6).
  • Summarized the decoupled representation (locomotion as metric camera motion, manipulation as each body's native skeleton, predicted jointly), the design principles (skeletons vs. mesh renderings; injection by spatial alignment), the limits of image-space interfaces and our extensions (relative depth, out-of-view context via the wide-FOV skeleton, wrist orientation via wrist-pose renderings), and five insights for downstream use.
  • These points are consolidated in a new "Findings" section (App. R) of the revised manuscript.

All revisions are highlighted in blue in the revised manuscript. Point-by-point responses are provided in each reviewer's discussion thread. We sincerely thank all reviewers again for their constructive suggestions and engagement throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.