acceptodds
Under review as a conference paper at ICLR 2027

EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos

11 views

Abstract

Egocentric human videos offer a scalable alternative to costly robot demonstrations, yet transferring their manipulation priors to robot control remains challenging. The commonly adopted retargeting-based approaches convert human wrist and finger poses into embodiment-specific action commands through hand-crafted geometric mappings. However, such mappings require meticulous recalibration across different embodiments and risk propagating hand-tracking errors into downstream robot control pipelines. To this end, we propose EgoJEPA, a three-stage framework that learns embodiment-agnostic representations from egocentric human videos without rule-based human-to-robot retargeting: 1) Human-action pre-training learns effect-aware latent actions through JEPA-style visual prediction conditioned on hand-motion differences; 2) Latent-action-based world modeling jointly predicts future visual dynamics and latent actions, which decouples human prior pre-training from robot-specific control parameterizations; 3) Embodiment-specific tuning adapts the pretrained model to diverse robots through lightweight action adapters. Experiments on five real-world manipulation tasks across three robot embodiments demonstrate the effectiveness and efficiency of EgoJEPA. It achieves a 74% average success rate under the out-of-domain evaluation, compared with 65% for the retargeting baseline, while reducing cumulative cross-embodiment wall-clock cost by 47.2%, from 743.2 to 392.7 hours.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Manipulation, JEPA, Egocentric Video

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_639XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EgoJEPA, a three-stage framework for transferring manipulation knowledge from egocentric human videos to robot control without explicit human-to-robot retargeting. Stage 1 learns latent action representations through JEPA-style future visual prediction conditioned on human hand-motion differences. Stage 2 trains a World-Action Model (WAM) to jointly predict future visual representations and latent actions. Stage 3 adapts the pretrained model to different robot embodiments using embodiment-specific action adapters and fine-tuning on robot demonstrations. Evaluated on five real-world manipulation tasks across three robot embodiments, EgoJEPA achieves 74% average out-of-domain success, compared with 65% for retargeting and 40% for robot-only training. The paper also investigates annotation robustness, robot-data efficiency, and cross-embodiment training costs.

Critical strengths
  1. A meaningful problem with a coherent practical approach. Learning transferable manipulation priors from human videos is an important direction for scalable robot learning. EgoJEPA provides a reasonable alternative to explicit human-to-robot retargeting by learning visually grounded latent actions and adapting them to robot-specific action spaces. The overall framework is coherent and addresses a practical limitation of existing approaches.
  2. Encouraging real-world manipulation results. The paper evaluates five real-world manipulation tasks across three robot embodiments, including dexterous hands and a gripper. EgoJEPA consistently outperforms the implemented baselines, with particularly strong improvements over robot-only training. The robustness and data-efficiency experiments provide additional evidence of practical value. I appreciate the emphasis on real-robot evaluation, which demonstrates the potential of the proposed framework beyond simulation.
Critical weaknesses
  1. Limited methodological novelty and insufficient validation of cross-embodiment transfer. The proposed framework mainly integrates existing techniques. Stage 1 uses a JEPA-style predictive objective with human hand-motion conditioning, Stage 2 follows the WAM paradigm with joint video-action flow matching, and Stage 3 introduces conventional action adapters. The main contribution lies in combining these components into a retargeting-free human-to-robot learning pipeline, rather than introducing a substantially new representation learning or world-modeling mechanism. The paper also provides limited discussion of closely related latent action learning methods, such as LAPO and Learning Latent Action World Models in the Wild, which have explored learning transferable action representations through future prediction. A clearer comparison with this line of work would help establish the distinctive contribution of EgoJEPA. Furthermore, each evaluated manipulation task is associated with a specific robot embodiment. While the results demonstrate that a shared pretrained model can be adapted to multiple robots, they do not directly establish transfer of the same manipulation skills across embodiments. The claimed cross-embodiment transferability would benefit from more controlled evaluation.
  2. Insufficient evidence isolating the contribution of JEPA-based latent action learning. A central claim is that predicting the visual effects of human motion produces more effective latent action representations than learning from motion alone. However, the current ablations do not sufficiently isolate this contribution. In particular, the JEPA and VQ-VAE comparison changes several factors simultaneously. EgoJEPA encodes temporal hand-motion differences and predicts future visual representations, whereas the VQ-VAE baseline reconstructs per-frame hand annotations using discrete latent representations. Therefore, the improvement from 63% to 74% cannot be attributed solely to JEPA-style visual prediction. A more controlled comparison using identical motion inputs, comparable encoder capacity, and different learning objectives would be more convincing. For example, a continuous motion autoencoder trained on the same hand-motion differences could help isolate the benefit of visual-effect supervision. The paper also provides limited direct evidence that the learned latent actions capture action-induced physical effects beyond conventional motion representations. Targeted representation analysis or probing experiments would help substantiate this claim. Additionally, the framework relies on strong pretrained Wan2.2 visual models, making it important to distinguish the benefits of the proposed latent action learning objective from those of the pretrained visual representations.
  3. Limited reproducibility and breadth of evaluation. The real-robot experiments are a valuable strength. However, the evaluation relies entirely on custom physical setups, making independent reproduction and comparison with other methods difficult. The large commercially acquired human-video dataset further raises the barrier to reproducing the complete training pipeline. Although the authors implement several relevant baselines, the absence of an established simulation benchmark or a publicly reproducible evaluation protocol limits broader comparison. A complementary simulation evaluation would be particularly useful for systematically comparing alternative latent action representations and training objectives under controlled conditions. The real-world results should remain the primary evidence of practical effectiveness, but a shared evaluation benchmark would significantly strengthen the paper's reproducibility and broader impact.
Questions for authors

Q1. Novelty and related work. How does EgoJEPA differ from existing latent action learning methods, particularly LAPO and Learning Latent Action World Models in the Wild? What specific advantages does human-motion-conditioned JEPA provide over simpler latent action representations? Q2. JEPA-specific contribution. Could the authors provide a more controlled comparison between JEPA-based visual prediction and motion reconstruction, using identical motion inputs and comparable representation architectures? What direct evidence supports the claimed effect-aware properties of the learned latent actions? Q3. Cross-embodiment transfer. Can the authors clarify the extent to which the experiments demonstrate transferable manipulation capabilities rather than the reuse of a shared pretrained model? Would evaluating the same manipulation skill across different robot embodiments provide stronger evidence? Q4. Reproducibility and broader evaluation. Could the authors provide complementary results on an established simulation benchmark or release a reproducible evaluation protocol? This would facilitate fair comparison with other latent action and policy learning methods while preserving the value of the real-robot results. EgoJEPA addresses an important problem and presents a coherent framework for transferring manipulation knowledge from human videos to robot control. The real-world evaluation is a notable strength, and the improvements over retargeting and robot-only baselines provide encouraging evidence of practical effectiveness. However, I have several reservations. The methodological contribution primarily lies in integrating existing techniques, while the specific advantages of JEPA-based latent action learning are not convincingly isolated. In particular, the VQ-VAE comparison changes multiple factors simultaneously, and the learned effect-aware representations receive limited direct validation. The cross-embodiment experiments also demonstrate model reuse more clearly than transferable manipulation skills. Furthermore, the exclusive reliance on custom real-robot evaluations limits independent reproducibility and broader comparison with alternative approaches. A complementary standardized evaluation would strengthen the experimental evidence. Overall, I lean toward weak rejection because the central representation-learning contribution remains insufficiently established. Nevertheless, I view the real-world results positively and would be open to raising my rating if the authors provide stronger controlled evidence for the benefits of JEPA-based latent actions and clarify the extent of cross-embodiment transfer.

Reviewer_JRQTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes EgoJEPA to transfer manipulation priors from human egocentric videos to multiple robot embodiments without rule-based retargeting. It follows a three-stage pipeline: learning an effect-aware latent action space via JEPA, building a latent world model using joint flow matching on a 5B video DiT, and adapting to downstream robots with lightweight MLPs. Real-world evaluations on three robotic platforms demonstrate solid out-of-domain success rates (74% vs. 65% baseline), reduced training time, and good robustness to tracking noise.

Critical strengths
  1. Extensive and Solid Real-World Robot Validation Across Heterogeneous Embodiments: Unlike many world model / manipulation studies that remain confined to simulation or single-gripper setups, this paper conducts thorough physical evaluations across three fundamentally different and high-dimensional embodiments (56-d Sharpa dexterous hand, 52-d Wuji industrial hand, and 14-d Xense Tac UMI gripper) over five contact-rich, multi-step tasks (e.g., cup stacking, zipper pulling, bottle unscrewing, and cardboard box folding). The physical experiments are well-documented, and the hardware engineering effort is commendable.

  2. Acutely Targeted Pain Points with Convincing Noise-Robustness Stress Tests: The paper directly tackles two prevalent bottlenecks in human-to-robot video transfer: tedious kinematic calibration and error amplification from noisy monocular hand tracking. The stress-testing protocol in Figure 6—injecting Gaussian position drift (up to 10 mm), temporal jitter (up to 12 frames), and contiguous annotation dropout (up to 40%)—is rigorous and provides convincing empirical evidence that predicting visual transitions in latent space acts as an effective semantic filter against non-physical tracking noise.

Critical weaknesses
  1. Misalignment Between Paper Nomenclature and Core Algorithmic Engine: The paper is titled "EgoJEPA", which naturally leads reviewers to anticipate an energy-based latent world model where downstream planning/control occurs in a non-generative JEPA representation space (akin to I-JEPA, V-JEPA, or RoboJEPA). However, in the actual system, the JEPA objective is strictly confined to Stage 1 as a representation pre-training pretext task; the Stage 1 predictor PϕP_\phi is entirely discarded thereafter. The core predictive and action-generation engine in Stages 2 and 3 is a generative diffusion/flow matching framework built upon a 5B video Transformer (Wan2.2 DiT). The submission would be more accurately categorized as a generative Flow-Matching World Action Model with a JEPA-regularized action space, and framing it primarily as a JEPA framework appears somewhat overstated.

  2. Nuance and Scope of the "47.2% Wall-Clock Time Reduction" Claim: The headline 47.2% reduction in training wall-clock time (from 743.2h to 392.7h across five tasks in Figure 7) is heavily influenced by the authors incorporating 150.9 hours of CPU-bound kinematic calibration into the retargeting baseline (e.g., 52.1h per new hand). When deploying on a single embodiment (Task 1: Sharpa), the time saving is modest (173.5h vs. 206.8h, a ~16.1% saving). The large cumulative savings only materialize under repeated multi-embodiment cold starts. The abstract and introduction should more transparently contextualize this metric rather than framing it as a universal end-to-end training acceleration.

  3. Reproducibility Barriers Regarding Commercial Data and Code Release: The pre-training relies on 20,106.6 hours of commercially licensed egocentric video with 3D hand keypoints, which cannot be released to the academic community due to vendor agreements. Furthermore, the repository linked on the project page is currently returning a 404 placeholder. Without open access to either the pre-training data or the codebase, independent verification of Stages 1 and 2 remains impossible for academic labs.

Questions for authors
  1. DiT Token Packaging & Loss Weighting in Stage 2: In Equation 5, how are the future visual latent tokens zτz_\tau and the 128-d action tokens uτu_\tau integrated into the Wan2.2 DiT? Are they concatenated along the token sequence dimension, or integrated via cross-attention / adaptive normalization? Furthermore, is there a balancing hyperparameter λ\lambda weighting the visual velocity loss against the action velocity loss?
  2. Missing Ablation on Action Pre-training Alternatives: Figure 3 compares EgoJEPA with EgoJEPA (VQ-VAE), demonstrating that predicting visual transitions is better than reconstructing isolated hand keypoints. However, what if Stage 1 is replaced by a standard inverse dynamics model (IDM) or a standard contrastive/predictive visual-action objective (e.g., LAPA or V-JEPA2 without hand差分 conditioning)? Does the empirical gain justify the added complexity of the 3-stage modular design over an end-to-end flow matching policy?
  3. CPU Calibration Overhead in the Retargeting Baseline: In Figure 7, the CPU-based calibration takes 52.1 hours per robot. In modern robot learning pipelines, once an embodiment's URDF and retargeting mapping are tuned, batch optimization or GPU-accelerated IK (e.g., cuRobo) can often compute keypoint-to-joint mappings significantly faster. Could the authors clarify why calibration required 52.1 hours for every new task/embodiment in their baseline setup?
  4. Reproducibility and Public Benchmark Commitments: Given that the 20k-hour dataset is commercial, do the authors plan to release a reproducible benchmark pipeline (and pre-trained checkpoints) trained on publicly available datasets like Ego4D or Assembly101? Will the complete training and adapter code be made publicly available upon publication?
Reviewer_kXNnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EgoJEPA, a three-stage framework for transferring manipulation priors from egocentric human videos to robots without rule-based human-to-robot retargeting. Stage 1 learns latent actions by predicting future visual representations conditioned on changes in annotated human hand motion; Stage 2 trains a world model to jointly predict future visual dynamics and those latent actions; Stage 3 adapts the model to robot action spaces using embodiment-specific adapters and robot demonstrations.

Critical strengths

The paper targets the embodiment-specific calibration and annotation-noise issues of retargeting. Learning action representations from their predicted visual consequences, then grounding them with lightweight robot-specific adapters, is a well-motivated design. The reported gains over the retargeting baseline and robot-only variant provide initial evidence that this separation can be useful.

Critical weaknesses

1.Evaluation covers five tasks, with each embodiment represented by a narrow set of tasks (one for Sharpa, three for Wuji, and one for the gripper). The out-of-domain setting tests object and spatial shifts, but does not establish transfer to new tasks or substantially different environments. Results are based on 20 trials per task, so success rates change in 5-point increments; the main results figure does not show uncertainty intervals. This limits confidence in the size and generality of the reported advantage. 2.The Stage 2-to-Stage 3 action-space transition is not specified clearly enough to assess or reproduce. Stage 2 predicts latent human-action targets, whereas Stage 3 supervises robot delta-actions. The text and Figure 2 describe input and output adapters that map between robot actions and the shared latent-action space, but Equation 9 does not make the input-adapter path explicit and writes the adapter as a composition after the world model. The paper also says the video and action Transformers are tuned in Stage 3, leaving unclear which pretrained components are preserved. This is central to the proposed transfer mechanism, not a minor implementation detail.

Questions for authors

1.Please specify the exact Stage 3 computation graph and loss: where are the input and output adapters applied, what are the dimensions and targets at each point, and which Stage 2 parameters are frozen or updated? How is robot-action supervision aligned with the latent-action representation learned in Stage 1? 2.Were the reported task success rates obtained from a single training run or multiple seeds? Please provide per-task counts and uncertainty intervals (or the underlying trial outcomes) for the main in- and out-of-domain comparisons. Could the 9-point out-of-domain difference from retargeting be distinguished reliably with this evaluation? 3.How are the cumulative wall-clock totals computed across tasks and embodiments—particularly which stages are reused for each baseline, and whether calibration, data processing, and failed or repeated runs are included? A precise accounting is needed to interpret the claimed 47.2% reduction. 4.What are the robot-training-data and optimization budgets for the external baselines, especially DexWild and LAPA, and are their results evaluated under the same trial protocol and test conditions? This affects how strongly the cross-method performance comparison supports the paper’s claims.

1 comment

After reading the authors' response, I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.