HetSkills: Reusable Heterogeneous Skill Learning for Physics-Based Control

We propose HetSkills, a versatile and extensible framework for physics-based character control that supports modular acquisition and composition of heterogeneous skills without retraining learned skills or the low-level controller. The key idea is to preserve learned skills as executable capabilities and further reuse them as control foundations, expert teachers, or behavioral priors for acquiring new skills. HetSkills begins by learning a tracking skill that establishes a strong foundation in motion control and creates a shared motion decoder, which can be reused across tasks without the need for retraining or separate controllers. To prevent the text-to-motion skill from exploiting shortcut pathways instead of learning language semantics, we introduce motion intuition distillation to ground text-to-motion generation in language semantics and a task-guidance module that dynamically adjusts actions based on high-level language instructions. This enables HetSkills to preserve natural motion while continuously expanding its skill repertoire, making it highly adaptable for long-horizon tasks. Experimental results demonstrate the effectiveness in motion tracking, text-to-motion generation, motion completion, and downstream task adaptation, achieving impressive success rates even under challenging conditions.