Unsupervised Latent Actions for Terrain-Aware Interactive Motion Generation

Generating responsive and terrain-aware character motion in real time is challenging due to the mismatch between high-frequency user inputs and the structured dynamics of human movement, as well as the need for stable closed-loop control. We present TINMO, a framework for Terrain-aware INteractive MOtion generation based on unsupervised Latent Action Codes. These discrete codes are learned directly from unlabeled motion data and represent high-level motion intentions, enabling real-time control without manual action annotations. TINMO models motion as transitions between spatiotemporal primitives indexed by Latent Action Codes, allowing coherent and efficient autoregressive generation. A lightweight multimodal controller maps user inputs, terrain features, and motion history to the latent codes and root trajectories, producing context-aware behavior in closed-loop settings. To support extensibility, TINMO includes an Action Library that stores new motion skills as action-code sequences and local trajectories for plug-and-play invocation through the frozen generator without retraining. Experiments on terrain-aware path-following tasks show that TINMO outperforms the evaluated baselines in motion tracking while supporting interactive responsiveness, demonstrating the effectiveness of unsupervised action modeling for real-time character control. Video demonstrations are available in the supplementary materials.