acceptodds
Withdrawn from ICLR 2027

Unsupervised Latent Actions for Terrain-Aware Interactive Motion Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating responsive and terrain-aware character motion in real time is challenging due to the mismatch between high-frequency user inputs and the structured dynamics of human movement, as well as the need for stable closed-loop control. We present TINMO, a framework for Terrain-aware INteractive MOtion generation based on unsupervised Latent Action Codes. These discrete codes are learned directly from unlabeled motion data and represent high-level motion intentions, enabling real-time control without manual action annotations. TINMO models motion as transitions between spatiotemporal primitives indexed by Latent Action Codes, allowing coherent and efficient autoregressive generation. A lightweight multimodal controller maps user inputs, terrain features, and motion history to the latent codes and root trajectories, producing context-aware behavior in closed-loop settings. To support extensibility, TINMO includes an Action Library that stores new motion skills as action-code sequences and local trajectories for plug-and-play invocation through the frozen generator without retraining. Experiments on terrain-aware path-following tasks show that TINMO outperforms the evaluated baselines in motion tracking while supporting interactive responsiveness, demonstrating the effectiveness of unsupervised action modeling for real-time character control. Video demonstrations are available in the supplementary materials.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Motion Generation, Interactive Character Control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MJVHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper trains a motion controller that maps user input (keyboard and mouse) and terrain observations to character motion. It first uses a motion primitive VAE to encode motion primitives into latents. A second network, the Latent Action Model, then extracts a discrete action token from two consecutive latents, similar to the inverse dynamics models used in video world models. A diffusion denoiser generates the next latent conditioned on the action token, the root trajectory and the motion history, and the VAE decoder reconstructs the motion. A controller predicts the action token and root trajectory from the user input, the terrain heightmap and the motion history. On top of this framework, the paper introduces an Action Library that stores motions as sequences of action codes, which can be invoked by semantic keys. The method is evaluated on a terrain-aware path-following task on the ALS dataset.

Contributions (as stated by the authors)

Formulating terrain-aware interactive motion generation as a motion world model, whose state space is a primitive-level latent and whose action space is a discrete code learned without supervision. A lightweight multimodal controller that maps short windows of keyboard and mouse input, together with local terrain features, to discrete latent action codes and root trajectories. An Action Library that stores new behaviors as sequences of existing action codes and local trajectories, enabling semantic invocation of complete behaviors through the frozen generator without retraining.

Critical strengths

Contributions (as stated by the authors)

Formulating terrain-aware interactive motion generation as a motion world model, whose state space is a primitive-level latent and whose action space is a discrete code learned without supervision. A lightweight multimodal controller that maps short windows of keyboard and mouse input, together with local terrain features, to discrete latent action codes and root trajectories. An Action Library that stores new behaviors as sequences of existing action codes and local trajectories, enabling semantic invocation of complete behaviors through the frozen generator without retraining.

Critical weaknesses

The latent action model is not necessary for motion generation. The VAE latent is already an implicit motion feature, and the decoder can reconstruct smooth results conditioned on the past motion. Extracting another discrete token from two latents, in the style of inverse dynamics models from video world models, adds complexity without clear benefit. If discrete actions are required, VQ-VAE-based methods already provide them. The Action Library is a complex and unnecessary version of motion matching. Storing motions as action-code sequences and retrieving them at runtime does what motion matching (learned motion matching) already does, with an additional generation pipeline. The paper studies an old problem but achieves worse results than prior work. Training motion controllers is a long-studied problem, and the results do not improve on existing controllers. There are severe artifacts. The results show severe foot sliding and incorrect intersections with the ground in every motion. The terrain-aware claim is overstated. All grounds in the results are planes. What the paper actually shows is interaction with obstacles or ground at different height levels, not terrain. Interaction with different obstacles has also been well investigated. The framing of the formulation is inaccurate. The paper claims to import the (state, action, transition) formulation from video and 3D world models. However, these terms have been widely used in the motion and RL literature for more than ten years. Related work is missing. The following work on motion VAEs and autoregressive generation is not discussed: Character Controllers Using Motion VAEs Real-time Controllable Motion Transition for Characters Many other papers also train motion controllers, for example: Mode-Adaptive Neural Networks for Quadruped Motion Control Neural State Machine for Character-Scene Interactions Local Motion Phases for Learning Multi-Contact Character Movements Learned Motion Matching

Questions for authors

Could the authors explain why the Latent Action Model is necessary

Reviewer_KpaaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TINMO enables real-time terrain-aware interactive human motion generation using unsupervised discrete latent action codes. Rather than predicting individual poses, it models motion as transitions between spatiotemporal primitives, addressing the mismatch between frequent inputs and structured motion. A Motion Primitive Variational Autoencoder (MVAE) encodes translation-free motion primitives, while a Latent Action Model (LAM) learns discrete codes for transitions between latent states without action labels. A latent diffusion model generates subsequent primitives conditioned on these codes, motion history, and root trajectories. A multimodal controller maps keyboard/mouse inputs, local terrain heightmaps, and motion history to action codes and continuous root trajectories. Local pose synthesis is separated from global root translation, so the model can handle motion dynamics and spatial navigation independently. An Action Library stores code sequences and local trajectories, allowing additional skills to be invoked through the frozen generator without retraining. Experiments on ALS evaluate closed-loop path following, terrain interaction, and responsiveness against retrieval-, diffusion-, and physics-based baselines. Separate experiments assess Action Library reconstruction on held-out motion clips, alongside component ablations.

Critical strengths
  1. TINMO integrates latent action modeling, motion primitives, and multimodal terrain-aware control. Pose–trajectory decoupling separates motion synthesis from spatial control, while the Action Library supports new skills without retraining. This structured interface addresses both motion continuity and terrain-dependent navigation, two practical requirements of interactive character animation.
  2. Promising experimental results supported by ablation studies. TINMO reports a trajectory error of 0.19 m versus 0.66 m for Motion Matching, with 11.82 ms latency per generated frame. Removing motion primitives, pose–trajectory decoupling, or terrain input degrades motion quality or tracking performance, providing further evidence for the effectiveness of these components.
Critical weaknesses
  1. Despite LAM's central role, the paper lacks comparisons with continuous latent actions or direct motion-primitive prediction without LAM. Action Library reconstruction shows that codes can represent motion skills but does not isolate their contribution to the reported gains. This weakens evidence that the unsupervised discrete representation itself, rather than the surrounding architecture, explains the main improvements.
  2. Limited evaluation under live interactive control. Although computational and direction/gait response times are reported, closed-loop tests replay recorded controls, and the user study rates videos rather than gameplay. The main path-following benchmark uses only four held-out ALS sequences. While recorded-input tests establish closed-loop generation, they do not show how users would adapt to generated motion. Responsiveness and robustness under unpredictable or rapid commands therefore remain uncertain for latency-sensitive gaming applications.
Questions for authors
  1. Can the authors compare discrete latent action codes with continuous actions or a controller that predicts motion primitives without LAM, to isolate its benefits for motion quality, trajectory tracking, and control? This comparison would directly address the claimed value of discrete transition codes.
  2. How are the reported direction and gait responses within 300 and 198 ms measured, given the 400 ms primitive update interval? Do these figures indicate mean or maximum latency, and what happens when commands change repeatedly within one primitive?
Reviewer_a8nMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work presents TINMO, a terrain-aware interactive motion generation system, by combining motion primitive VAE and discrete latent action. The discrete latent action, along with motion history and future 3D trajectory, are used as conditions for diffusion model to generation motion primitive latent, which is then decoded into character motions. To allows gamepad signal control and adapt to surrounded terrain, a multimodal controller is used to project this signal into latent action and trajectory to feed into the generator. An Action Library supports invocation of additional example behaviors through the frozen generator. The integration is practically interesting, but I find the technical contribution incremental and the evidence for motion quality and comparative performance insufficient.

Critical strengths
  1. The paper tackles a interesting and relevant problem of responsive character control over varied terrain.
  2. Integrating user input, motion history, terrain observations and reusable behaviors into one system is useful. The demonstrations illustrate terrain-traversal capabilities.
Critical weaknesses
  1. The design combines established components: primitive latent diffusion and rollout training from DARTControl. ARDY is particularly relevant to the root/body separation and interactive autoregressive generation. A terrain-aware baseline trained on the same data and control signals is important to demonstrate the effectiveness of the proposed design.
  2. The motion quality seems to be poor. There're obvious and repeated foot skating, penetration and motion popping. This might be result from the trajectory-based terrain traversal interface. Accurate root tracking does not necessarily imply plausible body motion or stable contacts.
  3. Some of the metric computation and baseline implementations is not convincing. TINMO’s contact consistency is only 15.0%, compared with 29.8% for GT, while penetration depth among penetrating samples remains 86.9 mm. These results require clearer interpretation alongside the visible artifacts. In table 1, DartControl receive the fid of 45.30 comparing with AMDM's 8.93, which seem reveal the potential implementation problem for the baselines.
Questions for authors
  1. How does the approach compare with ARDY, Categorical Codebook Matching and a matched terrain-aware controller?
  2. Can the authors report terrain-relative contact-conditioned skating and primitive-boundary discontinuity metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.