acceptodds
Withdrawn from ICLR 2027

Demo-JEPA: Joint-Embedding Predictive Architecture for One-shot Cross-Embodiment Imitation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robotic imitation learning is often treated as reproducing demonstrated actions, but actions are inherently embodiment-specific. When demonstrations come from humans or robots with different morphology, kinematics, or action spaces, this action-centric view requires shared action spaces, heuristic retargeting, or large-scale multi-embodiment co-training. We instead view demonstrations as implicit specifications of future goals: the target agent should infer what state the demonstrator is trying to realize, rather than how the demonstrator executes it. We propose Demo-JEPA, a cross-embodiment imitation framework that decouples demonstration intent from embodiment-specific execution. Built on a JEPA-based world model, Demo-JEPA translates source visual demonstrations into target-compatible future latent trajectories in a shared predictive representation space. The target agent then uses these latent trajectories as subgoals and realizes them through planning under its own learned forward dynamics. Because Demo-JEPA avoids action-level correspondence and requires only visual demonstrations plus the target agent's own interaction experience, it supports flexible imitation across heterogeneous embodiments. Experiments on RLBench and real-world manipulation tasks show that Demo-JEPA matches specialized in-domain planners and generalizes to unseen tasks and embodiment configurations where prior methods fail.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, world model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1sKjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses cross-embodiment imitation learning. The authors argue that treating a demonstration as a sequence of actions to reproduce is the wrong formulation due to the difference in source and target robots. They propose Demo-JEPA, which translates source demonstrations into trajectories in a shared JEPA representation space, and then optimizes actions toward them in target robot's action space. Experiments on RLBench and real-world manipulation tasks are reported across three suites of increasing distribution shift.

Critical strengths
  1. It is a good reframing of the problem. A demonstration is not an optimal trajectory to reproduce considering the difference in source robot and target robot. It is a sound choice to operate in a JEPA latent space rather than pixel space.

  2. The authors recognize that source-target temporal misalignment is the main vulnerability, and propose engineering solutions to mitigate this issue.

  3. Experiments cover both simulation and the real world

Critical weaknesses
  1. It is unclear what the major contributions are. The world model is adopted from V-JEPA 2.1. The only newly learned component is the dreamer predictor. Once the latent goal is given, the rest is standard goal-conditioned latent MPC on the target.

  2. Scene-configuration mismatch is motivated but never evaluated. Although target-side initial conditions are varied, but source and target are never systematically placed in different configurations.

  3. Transfer is evaluated toward one target (Franka arm), and all three arms are similar 6 or 7 DoF manipulators, so the morphological gap tested is modest.

  4. Baseline coverage is thin: no VLA baseline and no target-only behavior-cloning baseline. So I cannot see what the source demonstration contributes over training on target data alone.

Questions for authors
  1. What does conditioning on the source demonstration contribute relative to target data alone?

  2. Since the simulation source demonstrations are generated by retargeted replay of the target's trajectories, does trajectory retargeting dominate all three simulation suites by construction rather than by privileged information?

Reviewer_19RqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Demo-JEPA, a JEPA-based framework for cross-embodiment robot imitation learning that learns to map visual demonstrations from a source robot to the future target states of a target robot within a latent representation space. A Dreamer Predictor utilizes aligned video trajectories of both the source and target robots to learn this translation mapping. By leveraging the action-conditioned world model of the target robot, action sequences are optimized via the Cross-Entropy Method (CEM) to achieve the predicted target states. The paper evaluates the transfer from a Sawyer to a Franka arm in the RLBench simulation environment, as well as the transfer from a UR5e to a Franka arm in real-world settings. The most significant performance improvements reported in the study are observed in zero-shot tasks: in the simulation environment, Demo-JEPA achieves an average success rate of 0.36, compared to 0.04 and 0.03 for VPP and XSkill, respectively; in the real-world environment, Demo-JEPA attains an average success rate of 0.25, whereas VPP and XSkill yield 0.00 and 0.05, respectively.

Critical strengths
  1. The research problem is highly significant. This paper addresses the embodiment gap in cross-embodiment imitation learning, proposing to approach the transfer from the perspective of task objectives rather than action alignment. The research motivation is clear, and the conceptual approach provides valuable insights.
  2. The experimental results are reasonably convincing. The study validates the proposed method in both simulated and real-world robotic environments. Notably, in zero-shot generalization tasks, it achieves substantial performance improvements compared to VPP and XSkill, demonstrating the method's strong potential in scenarios involving distribution shifts.
Critical weaknesses
  1. The method's scalability has notable limitations. The Dreamer Predictor requires time-aligned, cross-embodiment paired trajectories for training, causing data collection and alignment costs to increase with the number of robot types and tasks. Its inability to directly utilize large-scale, heterogeneous, unpaired robot data restricts its potential to expand into a generalized cross-embodiment learning framework.
  2. Experimental validation is insufficient. The paper evaluates very limited robot combinations and lacks transfer experiments across embodiments with significant morphological differences. The training data scale for the Stage 0 world model and whether it includes zero-shot test tasks remain unclear. Furthermore, without ablation studies on key modules like Action Co-Training and Adaptive Goal Updating, the current experiments inadequately support the claims regarding generalization and overall effectiveness.
Questions for authors
  1. The Dreamer Predictor relies on time-aligned, paired cross-embodiment trajectories. Can this method utilize unpaired heterogeneous robot data for training?
  2. What is the exact scale of the training data used for the Stage 0 world model? Are the zero-shot test tasks and their environmental configurations strictly excluded from the Stage 0, Stage 1, and Stage 2 training data?
  3. Can you provide ablation results excluding Action Co-Training and Adaptive Goal Updating?
  4. The Stage 2 training objective has a fundamental validity issue. The lack of clear correspondence between input action sequences and the latent goals generated by the Dreamer Predictor may cause the world model to ignore action information and learn incorrect dynamics, thereby undermining the foundation for subsequent CEM planning.
  5. Demo-JEPA performs significantly worse than VPP on Behavior Grounding tasks and achieves only a marginal average improvement in real-world robotic tasks. Can the authors explain this significant performance drop on seen tasks and further clarify the method's overall advantages over VPP?
Reviewer_6GsoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Demo-JEPA, a framework for one-shot cross-embodiment imitation from visual demonstrations. Given a source robot demonstration and the target robot current observation, a Dreamer Predictor infers target-compatible future goals in a shared JEPA representation space. An action-conditioned world model then supports action selection through Cross-Entropy Method planning. The framework additionally incorporates temporal perturbation during training and adaptive subgoal advancement during execution.

Experiments evaluate Sawyer-to-Franka transfer on 11 RLBench tasks and UR5e-to-Franka transfer on six real-world tasks. Compared with VPP and XSkill, Demo-JEPA performs better on the selected unseen-task evaluations, while achieving lower success rates on familiar tasks.

Critical strengths
  • A sensible formulation of cross-embodiment imitation. Translating demonstrations into target-compatible goals allows the target robot to choose actions under its own dynamics. This is a useful approach when directly reproducing the demonstrator execution is inappropriate.

  • Evaluation includes physical robots. The combination of simulation and real-world experiments provides meaningful evidence beyond a purely simulated demonstration. The results show a consistent generalization advantage over the principal external baselines on the selected unseen tasks.

  • Several informative analyses. The oracle-goal comparison, architectural ablations, and Demo-DP extension help investigate the proposed components. The additional VPP variants in the appendix also make a useful effort to address differences in data usage and cross-embodiment supervision.

Critical weaknesses
  • The novelty over existing world-model and cross-embodiment approaches is insufficiently established. Predictive representations, visual goal conditioning, and planning through learned dynamics are established ideas. The contribution appears to lie primarily in their particular integration through the Dreamer Predictor. The paper should more clearly distinguish this contribution from prior cross-embodiment goal-learning methods and recent WAMs, including DreamZero, Fast-WAM, and DiT4DiT. In particular, avoiding full video reconstruction during deployment is not unique to JEPA-based control.

  • The empirical advantage is narrower than the overall framing suggests. Demo-JEPA underperforms VPP on familiar tasks in both simulation and the real world. Its real-world bridging advantage over VPP is only approximately 2.5 percentage points—one additional successful rollout across the two tasks. Demo-DP also substantially outperforms Demo-JEPA on familiar and bridging real-world tasks. The strongest evidence therefore concerns the selected unseen-task setting, where absolute success rates remain modest (approximately 36% in simulation and 25% on physical robots)

  • Important aspects of experimental comparability remain unclear. Matching task and trajectory counts does not necessarily match supervision, pretrained representations, or computational budgets. Demo-JEPA benefits from temporally aligned source–target videos, world-model pretraining, and inference-time optimization. Although the appendix strengthens VPP with additional supervision, it does not fully isolate the contributions of JEPA representations, paired supervision, and planning. The task coverage of Stage 0 also needs clarification before the zero-shot claims can be assessed confidently.

  • The breadth and statistical support are limited. Simulation uses one benchmark, with author-defined evaluation subsets, and real-world evaluation contains only six tasks. Results are reported without confidence intervals or variation across training seeds. Moreover, the target remains Franka in both settings, so the experiments do not establish transfer to an unseen target embodiment.

Questions for authors

Novelty: What is the precise contribution beyond combining cross-embodiment goal prediction with JEPA-based planning? Please distinguish the formulation from prior visual goal-learning methods and modern WAMs, while accounting for differences between test-time demonstration conditioning and video-based continued training.

  • Data splits: Please provide the task coverage and overlap for every training stage, including Stage 0. Do any zero-shot evaluation tasks contribute trajectories to world-model training, video alignment, or hyperparameter selection?

  • Fairness of supervision: What paired or aligned source–target information is available to each baseline? How much of the improvement can be attributed to this supervision rather than the JEPA representation?

  • Value of planning: Given Demo-DP’s stronger performance on several evaluation groups, what evidence establishes when CEM planning is beneficial? Please also report deployment latency and clarify whether execution budgets and stopping criteria are comparable across methods.

  • Reliability and scope: Can the authors provide uncertainty estimates or repeated evaluations for the main results, particularly the small real-world bridging improvement? Please also clarify whether “zero-shot” refers to unseen tasks, configurations, or target embodiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.