acceptodds
Withdrawn from ICLR 2027

EgoX: Egocentric Cross-Embodiment Manipulation with Embodiment Dreaming

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

17 views

Abstract

Scaling robot manipulation requires learning from demonstrations collected across humans and heterogeneous robots. These sources cover diverse objects, scenes, tasks, and interaction strategies, but gaps in sensing, kinematics, motion patterns, and action spaces make it difficult to train a shared policy effectively. We introduce EgoX, an egocentric cross-embodiment framework for robot manipulation. Its central design is to share manipulation structure at the level of semantically corresponding modalities, while preserving embodiment-specific sensing and control interfaces. EgoX stores human and robot demonstrations as unified egocentric observation-action trajectories, pretrains a Modularized Cross-Embodiment Transformer with Embodiment Dreaming (MXT+) on the mixture, and fine-tunes it on target-embodiment demonstrations. MXT+ combines embodiment-specific modality tokenizers and action experts with a shared Transformer trunk. An auxiliary embodiment dreaming objective predicts future embodiment latents from egocentric visual and proprioceptive features, using exponential-moving-average copies of the modality encoders; this training-only branch encourages the trunk to encode embodiment-specific temporal structure without adding deployment-time computation. We evaluate EgoX on five manipulation tasks across five real robots (wheeled humanoid, legged humanoid, quadrupedal manipulator, two tabletop manipulation platforms) and one simulated humanoid, using human demonstrations as an additional pretraining source. Across in-distribution and out-of-distribution settings, cross-embodiment pretraining improves downstream manipulation, broader embodiment mixtures improve transfer, and embodiment dreaming provides additional gains on contact-rich tool-use tasks. Anonymous project website: egox-embodiment.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Cross-embodiment Learning, Latent World Modeling, Learning from Human Videos, Embodiment-aware Latent Dynamics, Contact-rich Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pBdyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EgoX, a cross-embodiment manipulation learning framework for heterogeneous robot and human demonstration data. The method achieves cross-embodiment joint pre-training and target task fine-tuning through a unified egocentric modal representation, a shared Transformer backbone, and embodiment-specific action experts. Additionally, the authors introduce Embodiment Dreaming, which enhances policy learning by predicting the future latent representations of the embodiment. The paper validates the approach on five categories of manipulation tasks across five real-world robots and one simulated robot, with experiments demonstrating that cross-embodiment pre-training and Embodiment Dreaming improve downstream manipulation performance.

Critical strengths
  1. The experimental coverage is extensive. The method is validated on five morphologically distinct real robots and one simulated robot, covering dexterous manipulation, tool use, and long-horizon tasks, providing broad empirical support for its effectiveness.
  2. Experiments demonstrate that jointly utilizing human and heterogeneous robot data improves downstream manipulation performance on target robots. Furthermore, Embodiment Dreaming yields additional gains in specific tasks, demonstrating the framework's practical value.
Critical weaknesses
  1. The methodological novelty is limited. EgoX primarily builds upon the existing MXT architecture. The proposed Embodiment Dreaming is essentially future latent prediction via an EMA Teacher, closely resembling existing predictive representation learning methods. Without direct comparisons to other temporal prediction auxiliary objectives, its unique technical contribution remains unjustified.
  2. The approach relies on a standard two-stage paradigm of cross-embodiment pre-training and target-task fine-tuning, failing to demonstrate that a unified policy can control multiple robots without fine-tuning. Furthermore, given the limited training data scale and the absence of systematic scaling experiments or standard simulation benchmark evaluations, the current evidence is insufficient to support its broad claims regarding general cross-embodiment learning.
Questions for authors
  1. What is the unique contribution of Embodiment Dreaming compared to existing future latent prediction or JEPA-like auxiliary objectives? Are there comparative experiments with other auxiliary predictive objectives?

  2. Can the pre-trained model directly control different robots without target-task fine-tuning, and what is the corresponding performance?

  3. How does model performance change as training data volume, task variety, and embodiment diversity increase? Are there relevant scaling experiments?

  4. Why was the method not evaluated on standard benchmarks like LIBERO or RoboCasa? How do the current experiments ensure a fair comparison with other cross-embodiment learning methods?

Reviewer_YTVkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoX learns from egocentric human, real-robot, and simulated demonstrations using semantically corresponding observation and action modalities. MXT+ combines embodiment-specific tokenizers and action experts with a shared Transformer and an EMA-supervised future-latent prediction objective. It is pretrained on mixed embodiments and finetuned on target tasks. Evaluation spans five tasks and thirteen task–embodiment pairs across five physical robots and one simulated humanoid.

Critical strengths
  1. The physical evaluation spans substantially different bodies and end effectors, including dexterous hands, grippers, and a fixed tool. This provides useful evidence that the modular interface can accommodate heterogeneous manipulation platforms.
  2. The pretraining-mixture and dreaming ablations go beyond demonstrations alone. In particular, the seven task–embodiment comparisons with and without dreaming provide evidence that the auxiliary objective can improve downstream task progress.
Critical weaknesses
  1. The contribution of embodiment diversity is not sufficiently isolated. Figure 4 omits target-embodiment-only pretraining: for example, pretraining on G1 data only and then finetuning on the G1 target-task data. The reported mixtures change data volume and task composition, and the full mixture additionally introduces the target robot's own non-target-task demonstrations. These comparisons support the usefulness of additional data but do not establish the advantage of heterogeneous pretraining over same-embodiment multitask pretraining. Matched-budget controls are needed, with data-volume controls to assess diversity itself.
  2. Downstream evaluation uses task-specific finetuning on each target embodiment. The paper does not report multitask performance from one checkpoint on a single embodiment, or from one shared trunk with embodiment-specific heads across multiple tasks and embodiments without further task-specific adaptation. Consequently, the breadth of separately adapted policies does not establish that the shared model retains and executes multiple skills jointly. How does performance change under single-embodiment multitask and multi-embodiment multitask post-training?
  3. Model size, visual-backbone identity, initialization before pretraining, and stage-specific training budgets are also unspecified. This prevents assessment of reproducibility, computational cost, and training-budget comparability across methods.
  4. The comparisons and evaluation protocols leave the methodological advantage unclear. External policy comparisons are limited to ACT trained on the target task and finetuned π0.5. Both are useful baselines, but neither represents an alternative predictive-policy approach. MXT provides a no-dreaming control; the advantage over existing predictive objectives remains untested. Given the connection to world-action modeling in Appendix A.1, FLARE and Fast-WAM are particularly relevant comparisons; Fast-WAM also removes future prediction at inference. A representative contemporary WAM, such as DreamZero, LingBot-VA, or a Cosmos 3 policy variant, would further contextualize performance, with adaptation and pretraining budgets disclosed.
  5. Simulation evaluation is confined to the authors' G1 kitchen setup. The substantial real-robot evaluation is valuable, but an established manipulation benchmark such as LIBERO, RoboTwin, or RoboCasa would enable reproducible external comparisons and test transfer beyond the custom environments. This is relevant given the framework's existing tabletop embodiments, although suitable observation/action adapters would be required.
  6. Finally, reporting only partial-credit task progress, with just seven real-world OOD trials per pair, leaves complete-task reliability uncertain; full-task success rates with confidence intervals are needed.
Questions for authors
  1. Does cross-embodiment pretraining outperform G1-only multitask pretraining followed by identical G1 task finetuning, with comparable training budgets? Does the advantage persist when pretraining data volume is controlled?
  2. Can you compare against a relevant predictive-policy baseline and evaluate transfer on an established manipulation benchmark, reporting complete-task success alongside progress scores?
  3. What are the parameter counts, visual-backbone and initialization sources, and pretraining/finetuning configurations and compute costs? How are the baseline and ablation training budgets matched?
Reviewer_ZkeKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the problem of benefitting from cross embodiment datasets for adapting to different target embodiments. The proposed method, Modularized Cross-Embodiment Transformer with Embodiment Dreaming (MXT+) breaks down action and states spaces into components shared across embodiments (eg. right hand action, left arm action), uses a shared architecture that features embodiment specific encoders and decoders for different components of the state and action spaces, and also predicts encodings of future states. The experiments feature five diverse real robots and one simulated humanoid, and show benefits of different amounts of pretraining data (including human demonstrations), and benefits of incorporating the future prediction objective.

Critical strengths
  1. The paper studies important problem of transferring from cross-embodiment datasets. The diversity of embodiments studied in the paper, from quadrapedal manipulators to hands with different number of fingers, is impressive. The paper includes all possible training sources -- human videos, simulated data, different real robot embodiments (beyond those studied in a single prior work like HPT [Wang et al. 2024], MXT [Niu et al., 2025] or CrossFormer [Doshi et al., 2025]).
  2. Along the lines of 1, the evaluation features diverse embodiments, including ID and OOD settings of multiple tasks, aiming to show benefits of cross-embodiment pretraining and future prediction.
Critical weaknesses
  1. The primary concern is the presented approach lacks methodological novelty. The method builds upon prior work, MXT [Niu et al., 2025], that already explores modularized action and state spaces that can be shared across embodiments. Modularizing actions is also popular in prior works that organize them into fixed dimension actions for training on multiple embodiments [1]. The method adds future prediction loss in relation to MXT, which is known to be helpful in general [2], also in context of cross embodiment generalization and new embodiment adaptation [3, 4].

  2. The experiments for the presented cross-embodiment transfer claims are not completely convincing.

  • Unseen embodiment generalization not studied as claimed. The target tasks and target embodiment are not completely unseen in the pretraining mixture used for the full method numbers (Figure 3). Results show that excluding target embodiment from the pretraining mixture hurts for most tasks in Figure 4, possibly indicating that the gains over no pretraining baseline could be coming from pretraining on a mixture that includes target data.
  • The presented comparisons to prior work only include π0.5\pi_{0.5} and ACT trained on target data, which cannot directly benefit from pretraining mixture. They do not include comparisons to prior cross embodiment works (CrossFormer [Doshi et al., 2025], HPT [Wang et al. 2024]) mid-trained or pretrained on the pretraining mixture used in the paper.

[1] RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation. Liu et al. ICLR 2025.

[2] Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation. Wu et al. ICLR 2024.

[3] World Action Models are Zero-shot Policies. Ye et al. 2026.

[4] Motubrain: An Advanced World Action Model for Robot Control. Xiang et al. 2026.

Questions for authors
  1. Would request authors to clarify contributions in relation to prior work. Please refer to Weakness 1 for the specific concerns.
  2. Can the authors provide additional details of the training and inference stages? Specifically:
  • How many steps are the pretraining and finetuning stages for each of the reported numbers in the paper?
  • How is checkpoint selected for evaluation?
  1. How do the method's numbers compare to a baseline that pretrains on the target data? Essentially if the proposed method MXT+ is pretraining on cross embodiment data for PP steps and target data for TT steps. Would we still see gains with MXT+ over a baseline that directly trains on target data for P+TP+T steps?
  2. Can the authors report complete success metrics with error bars in addition to the reported progress metrics? In addition, a breakdown or analysis of what task stages specifically improve with cross-embodiment training and embodiment dreaming would be useful to understand why cross-embodiment data helps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.