HSI-Agent: Bridging Motion Dynamics and Scene Semantics for Unified Human-Scene Interaction Generation and Understanding

Human-scene interaction (HSI) generation and understanding rely on the same knowledge of human motion, scene geometry, and object interactions. A unified model can share this knowledge and support richer interaction than task-specific motion generators. However, current approaches still face following limitations. First, generation and understanding remain separate prediction tasks, even when both are included in one model. They are not explicitly connected through scene-grounded plans and persistent human-scene states. Second, a shared representation of complete motion in world coordinates mixes local body dynamics with global scene placement. This representation causes large encoding errors and makes it difficult to use motion priors learned from diverse motion data. We propose HSI-Agent, a unified framework for HSI planning, generation, and understanding. A language planner and a scene planner convert open-ended instructions into ordered interaction plans and maintain the current human-scene state. We then introduce a factorized representation based on root-origin anchoring. Motion Tokens encode root-relative body dynamics from pretrained motion models, while Affordance Tokens encode target regions and scene constraints. These tokens provide a shared interface across the framework. For generation, Trajectory-Flow predicts global root trajectories that place motion priors in the scene. For understanding, separate streams recover motion and interaction semantics from the shared token encoders. Joint training further allows generation supervision to improve HSI understanding. Experiments on HUMANISE and augmented HumanML3D show state-of-the-art performance in HSI generation and understanding. Evaluation on LINGO and TRUMANS further demonstrates improved physical generalization across datasets.