- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents HSI-Agent, a unified framework for human–scene interaction planning, generation, and understanding. The method combines a language planner and a scene planner to decompose complex instructions into scene-grounded interaction plans. It introduces a factorized representation with Motion Tokens and Affordance Tokens to separate local body dynamics from global scene constraints. A Trajectory-Flow module generates scene-compatible root trajectories using pretrained motion priors, while a shared representation supports both motion generation and interaction understanding through joint training. Experiments on HUMANISE, augmented HumanML3D, LINGO, and TRUMANS demonstrate improvements in generation, understanding, and cross-dataset physical generalization.
Critical strengths
- Unified framework. Integrates HSI planning, motion generation, and interaction understanding within a single framework.
- Effective motion factorization. Separates local body dynamics from global scene placement, enabling the reuse of pretrained motion priors.
- Flexible interaction planning. Supports complex, multi-step instructions through language and scene-aware planning.
- Strong quantitative results. Achieves competitive or superior performance on HSI generation and understanding benchmarks.
- Cross-dataset generalization. Demonstrates improved physical plausibility on LINGO and TRUMANS.
Critical weaknesses
- Insufficient long-horizon evaluation. The paper lacks quantitative evaluation of multi-step instruction following and motion continuity. The referenced supplementary videos are also missing, making temporal quality difficult to assess.
- Limited scene adaptation. Trajectory-Flow only adjusts global root trajectories while preserving local articulated motion, potentially limiting adaptation to different object geometries and contact configurations.
- Trade-offs in physical plausibility. Although scene penetration decreases, contact accuracy and foot skating worsen on LINGO and TRUMANS, raising concerns about motion realism.
- Insufficient ablations. The paper lacks direct comparisons against simpler root-placement methods and ablations of key components such as Trajectory-Flow and motion stitching.
- Limited validation of unified learning. Joint training improves understanding, but its impact on generation is unclear. The end-to-end benefits of integrating planning, generation, and understanding remain insufficiently evaluated.
Questions for authors
Could the authors provide the referenced supplementary videos, particularly side-by-side motion comparisons and long-horizon interaction sequences? The author claims to have it in the paper, but I'm not able to locate it in the supplementary file section of OpenReview. Also, why does improved collision avoidance come with worse contact and foot-skating metrics? Can the authors further analyze this trade-off?