Tokens of Motion: Primitive Motion Learning for Human-Humanoid Skill Transfer

Human videos provide a scalable source of manipulation experience, yet transferring them to robots remains challenging because human demonstrations lack robot-compatible actions and differ substantially from robots in embodiment and kinematics. We propose Motion Primitives for Action Composition and Transfer (M-PACT), a framework that addresses this challenge by factorizing primitive motion learning from embodiment-specific motion realization. This separation allows us to learn generalizable and reusable motion primitives from large-scale 3D human hand-pose trajectories, while requiring only limited embodiment-specific robot data to ground these primitives into executable motions. Specifically, we first discover local motion segments from human hand trajectories and learn a compact primitive-motion representation that captures recurring manipulation patterns independently of robot embodiment. Given task and observation context, these primitives are then selected and composed, while a lightweight embodiment-specific decoder maps them to robot motions. Crucially, transferring to a new embodiment preserves the learned primitive representation and requires adapting only the realization model, substantially reducing the dependence on costly robot demonstrations. Experiments across EgoDex, HoloAssist, and HumanoidEveryday demonstrate cross-dataset and cross-embodiment transferability of proposed M-PACT, with lightweight decoder adaptation achieving strong humanoid motion reconstruction using as few as $\sim$100 robot demonstrations. These results establish factorized primitive learning as a scalable bridge from large-scale human motion data to humanoid manipulation.