- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
This paper trains a motion controller that maps user input (keyboard and mouse) and terrain observations to character motion. It first uses a motion primitive VAE to encode motion primitives into latents. A second network, the Latent Action Model, then extracts a discrete action token from two consecutive latents, similar to the inverse dynamics models used in video world models. A diffusion denoiser generates the next latent conditioned on the action token, the root trajectory and the motion history, and the VAE decoder reconstructs the motion. A controller predicts the action token and root trajectory from the user input, the terrain heightmap and the motion history. On top of this framework, the paper introduces an Action Library that stores motions as sequences of action codes, which can be invoked by semantic keys. The method is evaluated on a terrain-aware path-following task on the ALS dataset.
Contributions (as stated by the authors)
Formulating terrain-aware interactive motion generation as a motion world model, whose state space is a primitive-level latent and whose action space is a discrete code learned without supervision. A lightweight multimodal controller that maps short windows of keyboard and mouse input, together with local terrain features, to discrete latent action codes and root trajectories. An Action Library that stores new behaviors as sequences of existing action codes and local trajectories, enabling semantic invocation of complete behaviors through the frozen generator without retraining.
Critical strengths
Contributions (as stated by the authors)
Formulating terrain-aware interactive motion generation as a motion world model, whose state space is a primitive-level latent and whose action space is a discrete code learned without supervision. A lightweight multimodal controller that maps short windows of keyboard and mouse input, together with local terrain features, to discrete latent action codes and root trajectories. An Action Library that stores new behaviors as sequences of existing action codes and local trajectories, enabling semantic invocation of complete behaviors through the frozen generator without retraining.
Critical weaknesses
The latent action model is not necessary for motion generation. The VAE latent is already an implicit motion feature, and the decoder can reconstruct smooth results conditioned on the past motion. Extracting another discrete token from two latents, in the style of inverse dynamics models from video world models, adds complexity without clear benefit. If discrete actions are required, VQ-VAE-based methods already provide them. The Action Library is a complex and unnecessary version of motion matching. Storing motions as action-code sequences and retrieving them at runtime does what motion matching (learned motion matching) already does, with an additional generation pipeline. The paper studies an old problem but achieves worse results than prior work. Training motion controllers is a long-studied problem, and the results do not improve on existing controllers. There are severe artifacts. The results show severe foot sliding and incorrect intersections with the ground in every motion. The terrain-aware claim is overstated. All grounds in the results are planes. What the paper actually shows is interaction with obstacles or ground at different height levels, not terrain. Interaction with different obstacles has also been well investigated. The framing of the formulation is inaccurate. The paper claims to import the (state, action, transition) formulation from video and 3D world models. However, these terms have been widely used in the motion and RL literature for more than ten years. Related work is missing. The following work on motion VAEs and autoregressive generation is not discussed: Character Controllers Using Motion VAEs Real-time Controllable Motion Transition for Characters Many other papers also train motion controllers, for example: Mode-Adaptive Neural Networks for Quadruped Motion Control Neural State Machine for Character-Scene Interactions Local Motion Phases for Learning Multi-Contact Character Movements Learned Motion Matching
Questions for authors
Could the authors explain why the Latent Action Model is necessary