acceptodds
Withdrawn from ICLR 2027

Reference-Conditioned Cross-Embodiment Video Generation without 3D Robot Assets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human and robot demonstration videos provide a growing source of manipulation experience, yet each observation couples the interaction with the demonstrator's body. Reusing this experience across embodiments requires the target robot's configuration and occlusions to adapt to its morphology and the surrounding scene. This work presents Ego2Any, a reference-conditioned video diffusion model that generates corresponding realizations of egocentric manipulation interactions across robot embodiments. It represents the demonstrated interaction separately from target morphology, with morphology-conditioned temporal transport to infer target-specific body layouts throughout the interaction. These layouts guide trajectory-conditioned video synthesis to preserve the source interaction and scene context while remaining aligned with the source actions. Ego2Any supports embodiment changes within a shared model without requiring target meshes, kinematic models, or a separately prepared target initial frame at inference. We further construct EgoXPair, an action-paired dataset containing 1.0M retained videos across 10 target embodiments. Evaluations show that the generated demonstrations improve downstream policy learning, supporting cross-embodiment video generation as reusable manipulation experience. Our code and dataset will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
video generation, cross-embodiment learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JEduOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EGO2ANY, a reference-conditioned video diffusion model that retargets an egocentric manipulation video to another robot embodiment. Given the source video, its end-effector trajectory, camera intrinsics, and action-independent RGB references of the target robot, the model predicts a target-body layout over time and uses that layout to condition video synthesis. At inference it does not require target meshes, kinematic models, or a prepared target initial frame. Training pairs are obtained by retargeting source actions onto robot assets and compositing the renderings into the source scene. The resulting generator is used to build EGOXPAIR, with 1.0M retained videos from 100K source trajectories across 10 target embodiments. Experiments evaluate human-to-robot and robot-to-robot generation with a custom semantic score, and the effect of the generated videos on StarVLA policies in SIMPLER and LIBERO.

Critical strengths
  1. A shared inference interface, with the layout component isolated in a controlled study. EGO2ANY separates the source interaction from the target morphology. Morphology-conditioned temporal transport predicts the target-body layout from the source action and multi-view RGB references, and the video backbone is conditioned on that layout. Changing the references changes the embodiment within one model, without target meshes, kinematics, or a prepared initial frame at inference. Table 5 compares this design with LingBot-Control under the same training setup. Action conditioning alone moves embodiment from 15.2 to 17.3 and H2RCore from 21.4 to 23.6. Adding frame-wise body layout raises embodiment from 22.6 to 34.2 and H2RCore from 28.3 to 37.8; temporal transport and surface support bring the full model to 45.4. These results support a benefit from predicting the target-body layout, rather than an effect of adding an action channel alone.
Critical weaknesses
  1. The headline transfer score only partially measures whether the interaction is preserved. H2RCore and R2RCore place 80% of their weight on embodiment (Appendix A.3). An output with zero embodiment and perfect scene, action, goal, and temporal scores receives at most 20. The reported human-to-robot gain, from 21.8 to 45.4, is therefore carried mainly by embodiment: 16.7 for MiniMax H3 against 43.1 for EGO2ANY. On the interaction domains, VACE scores 95.0, 86.8, and 92.9 on action, goal, and scene, against 46.7, 44.4, and 71.8 for EGO2ANY. On robot-to-robot transfer, EGO2ANY raises embodiment from 9.4 to 50.0 over LingBot-Control, while scene falls from 80.7 to 72.6 and goal transition from 84.3 to 71.8. The domains are scored by Gemini 3.5 Flash, Qwen3.7-Plus, and GPT-6 Astra, with no reported agreement against human raters. A rendered geometric teacher is already available, but the main results do not report mask overlap or end-effector reprojection error on held-out composites. Those measurements would show whether the embodiment gain preserves the manipulation, which the weighted core does not establish by itself.
  2. The reported policy gains are not isolated from the amount of added data.
    The abstract and Figure 1 highlight gains of 13.4 and 12.7 points on SIMPLER. Those comparisons add the full EGOXPAIR corpus, 1.0M videos from 100K trajectories, to benchmark-only training. In the matched setting, which regenerates the benchmark’s own demonstrations, StarVLA-π moves from 58.7 to 64.5 and StarVLA-GR00T from 61.6 to 67.4. On LIBERO, both backbones improve by 1.4 points from baselines already near 95. The paired actions are the source end-effector commands. There is no comparison, at the same sample count and optimization budget, against same-embodiment repetition or against the Appendix C renderings. Without that control, the success-rate difference does not isolate the contribution of cross-embodiment generation.
Questions for authors

Transfer score: Have you computed the core with equal domain weights? On the held-out routes, what are the visible-mask IoU and TCP reprojection error against the rendered teacher? Can you also break out embodiment and action for Panda, UR5e, Rizon 4, and Unitree G1?

Downstream attribution: In the SIMPLER runs, were the optimization steps and the number of added samples matched? At the same sample count, how does matched EGO2ANY data compare with same-embodiment repetition and with the Appendix C composites?

Reviewer_LfwGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Ego2Any, a reference-conditioned video diffusion framework for transferring human and robot manipulation demonstrations across different robot embodiments. The method uses source videos, action trajectories, and target robot RGB references to predict target-specific body layouts and generate corresponding manipulation videos without requiring target 3D assets at inference time.

The authors also introduce EgoXPair, a dataset containing 1 million generated videos across 10 robot embodiments. Experiments demonstrate improvements over existing video generation baselines and show that augmenting VLA training with EgoXPair improves downstream policy performance on SIMPLER and LIBERO.

Critical strengths

The paper addresses an important problem of generating cross-embodiment robot demonstrations without requiring 3D robot assets at inference. The proposed method shows improvements over existing video generation baselines, and the generated data improves downstream VLA policy performance.

Critical weaknesses

W1. The paper mainly evaluates robot embodiments that are already included in the training data. It is unclear whether the model can generalize to completely unseen robots using only RGB references. The authors should provide experiments on unseen robot embodiments.

W2. The overall evaluation score assigns 80% of the weight to target embodiment, while task action, goal completion, and temporal consistency receive much lower weights. This may overemphasize robot appearance rather than correct manipulation. The authors should justify this weighting and provide evaluations with more balanced metrics.

W3. The paper shows that EgoXPair improves VLA policy performance, but it is unclear whether these improvements come from cross-embodiment transfer or simply from adding more training data. The authors should compare against equal-sized conventional data augmentation or same-embodiment synthetic datasets.

Questions for authors

I would appreciate it if the authors could address the concerns and questions raised in the Weaknesses section above.

Reviewer_AWtZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents Ego2Any, a reference-conditioned framework for translating manipulation videos across robot embodiments without requiring target 3D assets. It combines source-body removal, scene completion, target-body layout prediction, and video generation. The resulting EgoXPair dataset can be used to augment downstream policy training.

Critical strengths
  1. The problem is well motivated. The proposed framework enables reference-conditioned cross-embodiment video generation without requiring target robot 3D assets at inference, potentially reducing the cost of robot-specific demonstration synthesis.
  2. The morphology-conditioned temporal transport and reference-conditioned surface alignment provide structured spatial guidance for adapting source interactions to different robot embodiments.
  3. Experiments on SIMPLER and LIBERO demonstrate that the generated EgoXPair data can improve downstream policy learning in simulation.
Critical weaknesses
  1. The advantages of the proposed temporal transport and surface alignment over simpler geometry-guided video editing or conditional generation methods remain insufficiently established.
  2. The downstream evaluations focus on data augmentation rather than policy learning on unseen embodiments without real target-robot demonstrations. A more direct test would train a VLA policy using only demonstrations synthesized for the new embodiment and evaluate its task success on that embodiment.
  3. The visual quality of the appendix and supplementary visualizations is poor, with noticeable blur, ghosting, and boundary artifacts. These issues raise concerns about demonstration fidelity.
  4. Visual alignment with source TCP trajectories does not guarantee executable target-robot actions, valid contacts, or physical feasibility.
  5. The writing is difficult to follow, especially the method section, where the presentation does not clearly connect the pipeline stages, and important implementation details remain vague.
Questions for authors
  1. Can policies trained solely on synthesized demonstrations generalize to previously unseen robot embodiments without real target-robot data?
  2. How does the proposed architecture compare with simpler conditional video editing methods under matched training data and supervision?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.