acceptodds
Withdrawn from ICLR 2027

HSI-Agent: Bridging Motion Dynamics and Scene Semantics for Unified Human-Scene Interaction Generation and Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human-scene interaction (HSI) generation and understanding rely on the same knowledge of human motion, scene geometry, and object interactions. A unified model can share this knowledge and support richer interaction than task-specific motion generators. However, current approaches still face following limitations. First, generation and understanding remain separate prediction tasks, even when both are included in one model. They are not explicitly connected through scene-grounded plans and persistent human-scene states. Second, a shared representation of complete motion in world coordinates mixes local body dynamics with global scene placement. This representation causes large encoding errors and makes it difficult to use motion priors learned from diverse motion data. We propose HSI-Agent, a unified framework for HSI planning, generation, and understanding. A language planner and a scene planner convert open-ended instructions into ordered interaction plans and maintain the current human-scene state. We then introduce a factorized representation based on root-origin anchoring. Motion Tokens encode root-relative body dynamics from pretrained motion models, while Affordance Tokens encode target regions and scene constraints. These tokens provide a shared interface across the framework. For generation, Trajectory-Flow predicts global root trajectories that place motion priors in the scene. For understanding, separate streams recover motion and interaction semantics from the shared token encoders. Joint training further allows generation supervision to improve HSI understanding. Experiments on HUMANISE and augmented HumanML3D show state-of-the-art performance in HSI generation and understanding. Evaluation on LINGO and TRUMANS further demonstrates improved physical generalization across datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Scene Interaction, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xYG4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HSI-Agent, a unified framework for human–scene interaction planning, generation, and understanding. The method combines a language planner and a scene planner to decompose complex instructions into scene-grounded interaction plans. It introduces a factorized representation with Motion Tokens and Affordance Tokens to separate local body dynamics from global scene constraints. A Trajectory-Flow module generates scene-compatible root trajectories using pretrained motion priors, while a shared representation supports both motion generation and interaction understanding through joint training. Experiments on HUMANISE, augmented HumanML3D, LINGO, and TRUMANS demonstrate improvements in generation, understanding, and cross-dataset physical generalization.

Critical strengths
  • Unified framework. Integrates HSI planning, motion generation, and interaction understanding within a single framework.
  • Effective motion factorization. Separates local body dynamics from global scene placement, enabling the reuse of pretrained motion priors.
  • Flexible interaction planning. Supports complex, multi-step instructions through language and scene-aware planning.
  • Strong quantitative results. Achieves competitive or superior performance on HSI generation and understanding benchmarks.
  • Cross-dataset generalization. Demonstrates improved physical plausibility on LINGO and TRUMANS.
Critical weaknesses
  • Insufficient long-horizon evaluation. The paper lacks quantitative evaluation of multi-step instruction following and motion continuity. The referenced supplementary videos are also missing, making temporal quality difficult to assess.
  • Limited scene adaptation. Trajectory-Flow only adjusts global root trajectories while preserving local articulated motion, potentially limiting adaptation to different object geometries and contact configurations.
  • Trade-offs in physical plausibility. Although scene penetration decreases, contact accuracy and foot skating worsen on LINGO and TRUMANS, raising concerns about motion realism.
  • Insufficient ablations. The paper lacks direct comparisons against simpler root-placement methods and ablations of key components such as Trajectory-Flow and motion stitching.
  • Limited validation of unified learning. Joint training improves understanding, but its impact on generation is unclear. The end-to-end benefits of integrating planning, generation, and understanding remain insufficiently evaluated.
Questions for authors

Could the authors provide the referenced supplementary videos, particularly side-by-side motion comparisons and long-horizon interaction sequences? The author claims to have it in the paper, but I'm not able to locate it in the supplementary file section of OpenReview. Also, why does improved collision avoidance come with worse contact and foot-skating metrics? Can the authors further analyze this trade-off?

Reviewer_6h4KOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HSI-Agent, a unified framework for human-scene interaction (HSI) planning, generation, and understanding. A Language Planner and a Scene Planner convert natural-language instructions into action sequences with target objects and spatial constraints. For motion generation, the framework separates body motion dynamics from scene-dependent global placement using Motion Tokens and Affordance Tokens. Trajectory-Flow predicts scene-conditioned root trajectories through conditional flow matching, allowing pretrained motions to be grounded in 3D scenes. Multi-step motions are composed through VQ-VAE-based refinement and trajectory alignment. For understanding, two streams capture intrinsic motion semantics and scene-dependent interaction semantics. Shared motion and affordance encoders are jointly optimized using generation and captioning objectives. Experiments cover HSI generation and understanding on HUMANISE, motion captioning on augmented HumanML3D, and cross-dataset physical generalization on LINGO and TRUMANS, alongside representation analyses and ablation studies.

Critical strengths
  1. Well-motivated factorized motion generation. By combining pretrained root-relative motion priors with scene-conditioned trajectories from Trajectory-Flow, the framework separates body dynamics from global placement. Table 3 shows consistent benefits of root-origin anchoring across three motion codecs. This addresses an important challenge in scene-grounded motion generation.
  2. Promising results across generation and understanding. On HUMANISE, Goal Distance decreases from 0.182 (HSI-GPT) to 0.011 (HY-Motion) and 0.033 (MotionGPT3), with improvements in several captioning and recognition metrics. Token ablations and joint optimization further support the effectiveness of the shared Motion and Affordance Tokens across both tasks.
Critical weaknesses
  1. Insufficient evidence for the central methodological contribution. Trajectory-Flow is a key technical component, but its independent benefits are not adequately established. The main comparisons involve different architectures and pretrained motion priors, while Table 3 supports the representation design rather than directly validating Trajectory-Flow. Without controlled comparisons against simpler trajectory prediction or alignment methods using the same motion prior, it remains unclear whether Trajectory-Flow is necessary for the reported improvements. This substantially weakens the evidence for the paper's central methodological contribution.
  2. The claimed agent-level contributions are not sufficiently validated. The paper emphasizes unified planning, persistent planning context, and long-horizon interaction, yet these capabilities are mainly supported by qualitative demonstrations. There are no direct quantitative evaluations of planning accuracy, multi-step task completion, or motion continuity, nor ablations isolating the planner and persistent context. As a result, the experiments primarily establish task-level generation and understanding performance, without adequately demonstrating the additional value of the proposed agent framework. This leaves a major part of the paper's claimed contribution insufficiently supported.
Questions for authors
  1. What evidence demonstrates the advantage of Trajectory-Flow over simpler trajectory prediction or alignment methods under the same pretrained motion prior? This would help isolate its specific contribution.
  2. Is there quantitative evidence that the proposed planner and persistent planning context improve multi-step interaction performance over simpler alternatives? This would help establish the effectiveness of the agent-level components.
Reviewer_5gL5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HSI-Agent combines language-based planning with human-scene motion generation and understanding. A pretrained motion model supplies root-relative body motion, while Trajectory-Flow predicts its global translation and orientation in the scene. Generation and understanding share motion and affordance encoders. Experiments evaluate target placement, contact, captioning, and transfer to LINGO and TRUMANS.

Critical strengths

Separating articulated motion from scene placement is a useful way to reuse motion models when interaction data have limited action diversity. On HUMANISE, the MotionGPT3 version reduces goal distance from HSI-GPT's 0.182 to 0.033 and improves the contact score from 92.31% to 96.35%. Joint training also improves verb and object recognition by 2.56 and 2.89 percentage points, respectively.

Critical weaknesses

Major

  1. The main scene adaptation changes only global translation and orientation, preserving the supplied local joint motion. This limits adaptation to furniture height and precise hand or foot contacts. Cross-dataset evidence is mixed: LINGO contact falls from 92.29% to 84.87%, and TRUMANS penetration depth rises from 1.29 to 2.76 cm, although penetrated frames decrease. The experiments establish better placement more clearly than consistently better interaction quality.

  2. The added value over simpler uses of the same motion prior is not established. A matched comparison with trajectory fitting or scene-constrained optimization would help isolate the need for Trajectory-Flow. The paper also needs to separate the benefit of its planner from that of the pretrained motion generator. These comparisons matter because factorization, flow matching, and shared encoders individually follow established approaches.

Minor

Related work should discuss SIMS [1], which combines LLM script generation with text-, scene-, and goal-conditioned physics-based control. Different task and output settings may preclude a direct main-table comparison, but the overlap in planning and scene interaction deserves discussion.

[1] Wang et al. SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation. ICCV 2025. https://arxiv.org/abs/2411.19921

Questions for authors
  1. Evidence affecting the assessment: With the same pretrained motion model and scene inputs, how does Trajectory-Flow compare with simpler trajectory fitting or constrained optimization?
  2. Clarification: How does the system accommodate different furniture heights or required contact points while preserving local joint motion? Please distinguish what the method adjusts from what the motion prior must already provide.
  3. Clarification/evidence: Equation 16 translates each subsequent trajectory to align segment boundaries. How are target placement and contact preserved after this translation, and are they evaluated on the final composed sequences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.