- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EgoJEPA, a three-stage framework for transferring manipulation knowledge from egocentric human videos to robot control without explicit human-to-robot retargeting. Stage 1 learns latent action representations through JEPA-style future visual prediction conditioned on human hand-motion differences. Stage 2 trains a World-Action Model (WAM) to jointly predict future visual representations and latent actions. Stage 3 adapts the pretrained model to different robot embodiments using embodiment-specific action adapters and fine-tuning on robot demonstrations. Evaluated on five real-world manipulation tasks across three robot embodiments, EgoJEPA achieves 74% average out-of-domain success, compared with 65% for retargeting and 40% for robot-only training. The paper also investigates annotation robustness, robot-data efficiency, and cross-embodiment training costs.
Critical strengths
- A meaningful problem with a coherent practical approach. Learning transferable manipulation priors from human videos is an important direction for scalable robot learning. EgoJEPA provides a reasonable alternative to explicit human-to-robot retargeting by learning visually grounded latent actions and adapting them to robot-specific action spaces. The overall framework is coherent and addresses a practical limitation of existing approaches.
- Encouraging real-world manipulation results. The paper evaluates five real-world manipulation tasks across three robot embodiments, including dexterous hands and a gripper. EgoJEPA consistently outperforms the implemented baselines, with particularly strong improvements over robot-only training. The robustness and data-efficiency experiments provide additional evidence of practical value. I appreciate the emphasis on real-robot evaluation, which demonstrates the potential of the proposed framework beyond simulation.
Critical weaknesses
- Limited methodological novelty and insufficient validation of cross-embodiment transfer. The proposed framework mainly integrates existing techniques. Stage 1 uses a JEPA-style predictive objective with human hand-motion conditioning, Stage 2 follows the WAM paradigm with joint video-action flow matching, and Stage 3 introduces conventional action adapters. The main contribution lies in combining these components into a retargeting-free human-to-robot learning pipeline, rather than introducing a substantially new representation learning or world-modeling mechanism. The paper also provides limited discussion of closely related latent action learning methods, such as LAPO and Learning Latent Action World Models in the Wild, which have explored learning transferable action representations through future prediction. A clearer comparison with this line of work would help establish the distinctive contribution of EgoJEPA. Furthermore, each evaluated manipulation task is associated with a specific robot embodiment. While the results demonstrate that a shared pretrained model can be adapted to multiple robots, they do not directly establish transfer of the same manipulation skills across embodiments. The claimed cross-embodiment transferability would benefit from more controlled evaluation.
- Insufficient evidence isolating the contribution of JEPA-based latent action learning. A central claim is that predicting the visual effects of human motion produces more effective latent action representations than learning from motion alone. However, the current ablations do not sufficiently isolate this contribution. In particular, the JEPA and VQ-VAE comparison changes several factors simultaneously. EgoJEPA encodes temporal hand-motion differences and predicts future visual representations, whereas the VQ-VAE baseline reconstructs per-frame hand annotations using discrete latent representations. Therefore, the improvement from 63% to 74% cannot be attributed solely to JEPA-style visual prediction. A more controlled comparison using identical motion inputs, comparable encoder capacity, and different learning objectives would be more convincing. For example, a continuous motion autoencoder trained on the same hand-motion differences could help isolate the benefit of visual-effect supervision. The paper also provides limited direct evidence that the learned latent actions capture action-induced physical effects beyond conventional motion representations. Targeted representation analysis or probing experiments would help substantiate this claim. Additionally, the framework relies on strong pretrained Wan2.2 visual models, making it important to distinguish the benefits of the proposed latent action learning objective from those of the pretrained visual representations.
- Limited reproducibility and breadth of evaluation. The real-robot experiments are a valuable strength. However, the evaluation relies entirely on custom physical setups, making independent reproduction and comparison with other methods difficult. The large commercially acquired human-video dataset further raises the barrier to reproducing the complete training pipeline. Although the authors implement several relevant baselines, the absence of an established simulation benchmark or a publicly reproducible evaluation protocol limits broader comparison. A complementary simulation evaluation would be particularly useful for systematically comparing alternative latent action representations and training objectives under controlled conditions. The real-world results should remain the primary evidence of practical effectiveness, but a shared evaluation benchmark would significantly strengthen the paper's reproducibility and broader impact.
Questions for authors
Q1. Novelty and related work. How does EgoJEPA differ from existing latent action learning methods, particularly LAPO and Learning Latent Action World Models in the Wild? What specific advantages does human-motion-conditioned JEPA provide over simpler latent action representations? Q2. JEPA-specific contribution. Could the authors provide a more controlled comparison between JEPA-based visual prediction and motion reconstruction, using identical motion inputs and comparable representation architectures? What direct evidence supports the claimed effect-aware properties of the learned latent actions? Q3. Cross-embodiment transfer. Can the authors clarify the extent to which the experiments demonstrate transferable manipulation capabilities rather than the reuse of a shared pretrained model? Would evaluating the same manipulation skill across different robot embodiments provide stronger evidence? Q4. Reproducibility and broader evaluation. Could the authors provide complementary results on an established simulation benchmark or release a reproducible evaluation protocol? This would facilitate fair comparison with other latent action and policy learning methods while preserving the value of the real-robot results. EgoJEPA addresses an important problem and presents a coherent framework for transferring manipulation knowledge from human videos to robot control. The real-world evaluation is a notable strength, and the improvements over retargeting and robot-only baselines provide encouraging evidence of practical effectiveness. However, I have several reservations. The methodological contribution primarily lies in integrating existing techniques, while the specific advantages of JEPA-based latent action learning are not convincingly isolated. In particular, the VQ-VAE comparison changes multiple factors simultaneously, and the learned effect-aware representations receive limited direct validation. The cross-embodiment experiments also demonstrate model reuse more clearly than transferable manipulation skills. Furthermore, the exclusive reliance on custom real-robot evaluations limits independent reproducibility and broader comparison with alternative approaches. A complementary standardized evaluation would strengthen the experimental evidence. Overall, I lean toward weak rejection because the central representation-learning contribution remains insufficiently established. Nevertheless, I view the real-world results positively and would be open to raising my rating if the authors provide stronger controlled evidence for the benefits of JEPA-based latent actions and clarify the extent of cross-embodiment transfer.
After reading the authors' response, I maintain my original score.