- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces an ego-centric loco-manipulation world foundation model with embodiments spanning across humans (10k hours) and robots (3k hours). An unified kinematic representation encodes locomotion through camera poses and manipulation through projected skeletons. It also introduces a long-horizon loco-manipulation benchmark based on dexterous humanoid teleoperation data. Evaluations show state-of-the-art performance on several benchmarks, ranging from zero-shot inference to few-shot and full-data post-training.
Critical strengths
This paper identifies a clear gap in current experimental robotics: lack of high quality curated datasets and controlled benchmarks, spanning multiple embodiments and domains, and proposes a complete pipeline to address this gap. It contains human-in-the-loop for quality checks, temporally aligned camera trajectories, captions, long-horizon dexterous teleoperation with locomotion and manipulation, etc. The contributions are straightforward and the paper is clear about that. Ablations are well positioned and clearly show the benefits of each design choice.
Critical weaknesses
The main weakness of the paper, in my opinion, is the lack of theoretical contributions, given that the targeted venue is ICLR. As I said above, the paper is clear about its contributions and they are relevant to the community, however I don't think there is enough novelty in each component individually: skeleton projection is well-known, plucker-ray maps are standard, so are spatially-aligned conditioning, human pretraining + human-robot (or robot) mid-training is a common recipe, and DMD for real-time interactive rollouts are common place at this point. Improvements in different benchmarks is expected given enough engineering effort and high quality datasets, which again, is worthy contribution in an of itself, but does not bring theoretical insights.
Questions for authors
Broadly speaking, I would like to hear from the authors what are the insights one could gain from this study, that could guide the development of new solutions to these problems (in particular the downstream applications mentioned in Section 5).
3 comments
Response to Reviewer dCJn (2/2)
3. Design principle: unlocking foundation-model capabilities through a simple interface.
- Simple, architecture-agnostic interface. We deliberately avoid specialized modules: a simple, effective interface unlocks the capabilities of already-strong video foundation models, so stronger base models translate directly into better world models. Scaling from 5B to A14B improves prediction in every regime: +2.2 dB on the cross-embodiment set (Table S15), +2.3 dB in-distribution on AgiBot (Table S14), and, without post-training, +2.1/1.7/1.4 dB on /-HOME/Xp-Teleop (Table 3 of the paper).
- Skeletons over mesh renderings. We weighed richer renderings such as URDF mesh masks and chose skeletons: a mesh also carries shape and contour—useful for a single robot but embodiment-specific—whereas a skeleton keeps only the kinematic structure, which human hands and robot FK keypoints share in form; human video can therefore pre-train the same interface directly, and human-video-only pre-training already follows unseen robots' camera motion zero-shot (Table S6; arm following in Figs. S16–S17).
- Injection follows spatial alignment. We also compared how each condition should enter the backbone: pixel-aligned camera (Plücker), skeleton, and wrist-pose maps are fused directly with the latents—addition and concatenation perform comparably (Table S19), as expected since a linear layer on concatenated inputs decomposes into a sum of per-input projections—whereas the wide-FOV skeleton extends beyond the frame and is not pixel-aligned, so it enters via cross-attention.
4. Limits of image-space interfaces, and how we extend them. A projected skeleton describes only what lies inside the image; we address three gaps that prior work leaves undiscussed:
- Depth ambiguity. For robots the skeleton comes from forward kinematics, so it stays accurate even under occlusion; what cannot be known in advance is the relative depth between the manipulator and the scene, and the model learns to resolve this ambiguity from large-scale human and robot data.
- Out-of-view context. An arm outside the frame still determines what enters the view, but an in-frame skeleton cannot represent it; since the FK skeleton remains accurate beyond the field of view, we render it with a wider field of view (1.5×). On RoboTwin 2.0, where the in-frame skeleton touches the image border in 99.4% of frames, this raises manipulator coverage from 0.37 to 0.56 and recovers the off-frame arm (Figs. S35–S36).
- Orientation ambiguity. A projected skeleton leaves the wrist's 3D orientation ambiguous; wrist-pose renderings supply the missing orientation explicitly.
5. Insights for downstream use.
- One evaluator across embodiments. Since conditions depend only on URDF and calibration and can be computed from commands (Table S8), the same interface and architecture evaluate on an arm and on a humanoid, with no per-robot action head or encoder.
- Relative rather than absolute judgments. Predicted and real outcomes are highly correlated in all three settings—task progress on RoboTwin (r = 0.984) and on the real G1 (r = 0.996 with the teleoperation reference, 0.991 without), and success rates on SIMPLE (r = 0.963)—with identical checkpoint rankings (MMRV = 0) even where absolute values differ (Fig. 4, Table S7, App. K); the model is thus best used for checkpoint selection and candidate comparison.
- Planning over multiple policies. Value-model selection over world-model rollouts consistently outperforms the best fixed checkpoint (92 vs. 85% task progress on RoboTwin, 55 vs. 45% on the real G1, and 80 vs. 70% success on SIMPLE; Fig. 4(c–d), Fig. S19), suggesting the world model exploits the complementary strengths of different policies.
- Priors pay off in generalization. As FastWAM's backbone, our model barely changes in-distribution LIBERO (96.95 → 97.00%) but improves LIBERO-Plus (50.06 → 56.82%), every few-shot setting (Table S13), and real-G1 task progress (33.3 → 56.7%, Fig. S20). World models should thus be judged by downstream utility, not video fidelity alone.
- Live loco-manipulation with humans as an embodiment. Prior live teleoperation of world models (e.g., DreamDojo) controls only the robot's upper body; in Xperience-0, the VR user's head motion drives the egocentric camera and hand motion drives the skeleton, so one model lets the user act as a human or a robot in real time (21.3 fps) over minute-long rollouts (Fig. S10).
We have consolidated these points in a new Findings section (App. R) of the revised manuscript, and we are committed to supporting the community through open-sourcing. We would be glad to discuss any of these points further.
Follow-up on rebuttal
Dear Reviewer dCJn,
With the discussion deadline approaching, we would appreciate your feedback on our rebuttal and revised manuscript at your earliest convenience. We have provided additional evidence and clarifications in response to your comments, and would particularly welcome an indication of which concerns, if any, remain unresolved, so that we have time to address them before the discussion closes.
Thank you again for your time and consideration.
Best regards,
The Authors
Response to Reviewer dCJn (1/2)
We sincerely thank Reviewer dCJn for recognizing our data pipeline, quality control, and benchmark as meaningful contributions, and the ablations as well positioned and clearly showing the benefit of each design choice. Added content in the revised paper is marked in blue.
A: Our contribution is an integrated framework: a unified kinematic interface that, to our knowledge, is the first to let one world model predict egocentric loco-manipulation across humans and robots, together with the data, benchmark, and insights that make this possible. The framework is not a simple assembly of existing techniques: its key choices are made against alternatives, the gaps of the chosen representation are identified and addressed, and substantial data work makes the geometry correct. Below we give the positioning, the central insight with its evidence, the design principles that follow, the limits of image-space interfaces and our extensions, and the insights for downstream use.
1. Positioning.
2. Central insight: a simple geometric interface is sufficient for cross-embodiment world modeling.