acceptodds
Under review as a conference paper at ICLR 2027

Tokens of Motion: Primitive Motion Learning for Human-Humanoid Skill Transfer

Abstract

Human videos provide a scalable source of manipulation experience, yet transferring them to robots remains challenging because human demonstrations lack robot-compatible actions and differ substantially from robots in embodiment and kinematics. We propose Motion Primitives for Action Composition and Transfer (M-PACT), a framework that addresses this challenge by factorizing primitive motion learning from embodiment-specific motion realization. This separation allows us to learn generalizable and reusable motion primitives from large-scale 3D human hand-pose trajectories, while requiring only limited embodiment-specific robot data to ground these primitives into executable motions. Specifically, we first discover local motion segments from human hand trajectories and learn a compact primitive-motion representation that captures recurring manipulation patterns independently of robot embodiment. Given task and observation context, these primitives are then selected and composed, while a lightweight embodiment-specific decoder maps them to robot motions. Crucially, transferring to a new embodiment preserves the learned primitive representation and requires adapting only the realization model, substantially reducing the dependence on costly robot demonstrations. Experiments across EgoDex, HoloAssist, and HumanoidEveryday demonstrate cross-dataset and cross-embodiment transferability of proposed M-PACT, with lightweight decoder adaptation achieving strong humanoid motion reconstruction using as few as 100 robot demonstrations. These results establish factorized primitive learning as a scalable bridge from large-scale human motion data to humanoid manipulation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Imitation Learning, Cross-Embodiment Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TY2LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents M-PACT, short for Motion Primitives for Action Composition and Transfer. It transfers manipulation skills from human demonstrations to humanoid robots through embodiment-independent motion primitives.

Critical strengths

The framework splits reusable motion primitives learned from human demonstrations from robot-specific execution. This lets it tap large human-motion datasets without demanding embodiment-specific robot demos at scale.

It runs four evaluations: primitive reconstruction, cross-dataset reuse, target-embodiment adaptation, and contextual primitive prediction. Error analysis on prediction and execution shows what the representation can and cannot do.

Critical weaknesses

Most experiments measure motion reconstruction error and primitive prediction accuracy, plus qualitative real-robot demos. The paper gives no systematic quantitative real-robot task success, and no head-to-head comparison with existing human-to-robot transfer methods on matched tasks and hardware.

Good trajectory reconstruction doesn't guarantee successful manipulation, especially where contact, grasp stability, and interaction dynamics matter. As it stands, the experiments don't really establish that cross-embodiment skill transfer works.

Questions for authors

see weakness

Reviewer_Ke9GOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes M-PACT, a motion primitive learning framework for human-to-humanoid skill transfer. The method segments human hand trajectories via SkillSeg, encodes them into translation, rotation, and gripper primitives using EPMT, and achieves motion transfer through lightweight embodiment-specific decoders. Additionally, CC-EPMT predicts these primitives from vision, language, and state inputs for closed-loop execution. Evaluations on the EgoDex, HoloAssist, and HumanoidEveryday datasets validate its motion reconstruction, cross-dataset representation reuse, and decoder adaptation capabilities, supplemented by qualitative demonstrations on real humanoid robots.

Critical strengths
  1. The modular design is valuable. Decoupling general motion primitive learning from embodiment-specific decoding provides a lightweight approach for human motion data reuse and robot adaptation.
  2. Cross-dataset representation transfer is effective. Evaluations on EgoDex, HoloAssist, and HumanoidEveryday validate motion primitive reconstruction and transfer, demonstrating the potential to reuse pre-trained representations by adapting only a lightweight decoder.
Critical weaknesses
  1. End-to-end skill transfer validation is inadequate. The evaluation primarily focuses on motion reconstruction error and token prediction accuracy. Real-world robot experiments lack quantitative task success rates and strong baseline comparisons, failing to sufficiently demonstrate effective human-to-humanoid skill transfer.
  2. Methodological novelty is limited, and the necessity of the core designs is questionable. SkillSeg, action tokenization, and embodiment-specific decoders are primarily combinations of existing techniques. Without thorough comparisons to existing action representation and motion retargeting methods, the paper fails to prove that factorized motion primitives significantly improve downstream robot manipulation performance.
Questions for authors
  1. Can you provide quantitative success rates for real-world robot tasks and include comparisons with existing human-to-robot transfer methods under identical conditions?
  2. Why did the deployed controller not utilize the full CC-EPMT and Human-Grounded Decoder? Can you provide closed-loop execution results for the complete framework?
  3. Can you include fair comparisons with action tokenization methods like VQ-BeT and FAST, as well as traditional motion retargeting approaches, to demonstrate the practical advantages of the proposed motion primitives?
  4. Given the claim that only ~100 robot demonstrations are required, can you provide end-to-end task success rates across varying data scales—rather than just reconstruction errors—to validate data efficiency and cross-embodiment transfer capabilities?
Reviewer_hZEWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

M-PACT learns motion primitives from 3D human hand trajectories for humanoid manipulation. SkillSeg segments motion, and EPMT represents each segment using discrete translation/aperture codes and a continuous rotation latent. A compact robot-specific decoder predicts endpoint effects, while context-conditioned predictors select primitives for execution. Evaluation covers offline reconstruction, primitive prediction, decoder adaptation, and qualitative real-G1 demonstrations.

Critical strengths
  1. Keeping the human-trained motion representation fixed while fitting a 27,399-parameter robot decoder provides a concrete approach to separating motion representation from embodiment-specific realization.
  2. Table S1 provides useful external representation controls: EPMT improves translation reconstruction over k-means at the same four-bit translation budget. This supports a component-level benefit beyond internal ablations.
Critical weaknesses
  1. The evaluation does not establish an end-to-end advantage for human-to-humanoid skill transfer. Real-robot results are qualitative, without task success rates, repeated-trial statistics, or external policy comparisons. The translation-only FAST/k-means comparison in Table S1 cannot establish superiority as a manipulation policy. Adapted MimicPlay or VQ-BeT, evaluated with matched data and execution interfaces, would be relevant comparisons.
  2. Many demonstrations are short closures, pushes, or grasps; although multistep examples exist, there is no quantitative evaluation of novel skill compositions supporting the introduction's claim of reuse across tasks and environments.
  3. The evaluated components also do not form one validated pipeline. Appendix 3.2 explicitly states that deployed controllers use neither the shared-trunk CC-EPMT objective nor the human-grounded decoder. Their offline results therefore cannot directly explain real-robot performance. Furthermore, the approximately 100-demonstration result concerns decoder reconstruction; primitive selection remains near the majority baseline through 250 episodes (Appendix 3.1.2). Data-efficient autonomous transfer remains unestablished.
  4. The necessity of learned segmentation and factorized primitives is insufficiently demonstrated. Table 2 evaluates each segmentation method on its own segments and endpoint targets, so lower reconstruction error does not isolate better segmentation from differences in target difficulty. Its learned row is also a single-seed precursor rather than the final seven-network SkillSeg ensemble. A comparison on shared trajectories or robot tasks against fixed-window segmentation is needed to justify this complexity.
  5. Moreover, EPMT encodes within-segment trajectories but supervises only their seven-dimensional net effects (Sec. 3.4); execution then interpolates to predicted endpoints. The experiments do not establish what this captures beyond directly predicting or quantizing endpoint changes, particularly for the short tasks demonstrated. Table S2 further shows that joint translation/aperture encoding reconstructs better than the factorized representation. Factorization may enable useful recombination, but its claimed practical value needs evidence from controlled composition or transfer experiments rather than reconstruction alone.
Questions for authors
  1. How does closed-loop task success compare with an external policy and a simpler direct-endpoint policy under matched training data and execution conditions, including tasks requiring previously unseen primitive combinations?
  2. Does the final SkillSeg ensemble improve task success over fixed windows, and does factorization improve held-out composition over joint encoding?
  3. What models and training data are used in each deployed controller? Which quantitative results apply to those exact configurations, and how does complete-policy performance vary with the number of robot demonstrations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.