acceptodds
Withdrawn from ICLR 2027

World2HOSI: A Multi-Expert Harness for Long-Horizon Human-Object-Scene Interaction Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon human–object–scene interaction (HOSI) is central to embodied AI, animation, and game development. Existing human-object interaction (HOI) and human–scene interaction (HSI) models excel in their respective domains, but none alone covers tasks requiring multiple interaction skills. Training unified HOSI models is hindered by scarce and costly data. We introduce World2HOSI, a multi-expert orchestration harness that generates continuous HOSI data from 3D Gaussian Splatting (3DGS) scenes and natural language instructions. Our key insight is that long-horizon HOSI hinges not on a stronger single model but on orchestration. Given a shared, structured scene state and an orchestration harness grounded in it, existing experts can jointly accomplish tasks beyond the reach of any of them alone. World2HOSI realizes this insight as a closed loop around a persistent scene state built from decoupled background and object assets. A harness plans action stages over this state, invokes experts via their native conditioning interfaces, and writes each action's effects back, so that every invocation is grounded in the evolving world. To reduce physical artifacts, bounded residual optimization and contact-driven physics simulation further ensure the physical plausibility of human motion and object manipulation, respectively, while preserving interaction intent. Experiments and a perceptual user study show that World2HOSI achieves the best task completion and interaction quality on the evaluated HOSI tasks, while generating more diverse data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human-object interaction, human-scene interaction, human-object-scene interaction, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DzTPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This submission proposes an agentic harness, named Word2HOSI to generate human-object-scene interactions over long time horizons from a given 3D scene and a text prompt as input. It first introduces a 3D scene representation (Interactive Scene Package) and then uses an LLM to plan and compile activity scripts, and then coordinates experts to generate interaction motions. It also incorporates a physics-based refinement for interactions. Experiments on 10 scenes and 20 tasks show improvement over baseline approaches.

Critical strengths
  • Many of the generated human interactions over long time horizons, provided in the supplemental, seem reasonable.

  • The evaluation results (Table 1) show improved performance of task completion, interaction plausibility, and human motion quality, when compared to strong baselines like HOI-FHLI. The perceptual study (Figure 6) provides further support in terms of visual realism and physical plausibility.

Critical weaknesses
  • The main concern is the unclear technical innovations. The authors claim to design a multi-expert harness, but the scene representation (Section 3.1) and task planning (Section 3.2) seem straightforward to me. Cross-expert reconciliation (Section 3.3) also seems trivial, though improving the motion transition. I don't feel like I have learned a lot after reading this work.

  • With the increasing capabilities of frontier models like GPT-Astra, such a harness design is questionable. I am curious how well GPT-Astra works to generate the human interactions without this harness.

  • In the experiments, there is no robustness test of this harness against different expert alternatives (Section A.2.2). In fact, the proposed harness uses HOI-FHLI for dynamic object interaction generations, but it's unclear why it achieves better object motion performance in Table 1.

  • The interaction generation results shown in the supplemental still have noticeable issues like foot sliding and physical penetration (like the book stack handling). It is questionable whether the physics-based refinement is effective or not.

  • The ablation studies did not include justifications for the contributions listed in Section 1, including the structured scene-state representation and the closed-loop generation mechanism. For the ablation of bounded residual refinement, there is no corresponding elaboration in the methodology section.

  • The long horizon generations lack quantitative results.

  • Related Work (Section 2) lacks discussions of relevant works on harness designs for 3D/4D generation (although not directly on human interaction generation) and how this harness design differs from them at the conceptual level.

  • The writing needs improvement. The current version reads like LLM-generation heavy across sections, and several sections are hard to decipher, including Section 3.2 - Compilation and Section 3.4.

Questions for authors

My questions are mainly on the technical innovation and differentiation, unclear performance against frontier models, unclear gain over HOI-FHLI, and missing ablations, as outlined above in the Critical Weaknesses.

Reviewer_2cFBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents World2HOSI, a framework for generating text-guided, long-horizon human-object-scene interactions in reconstructed 3DGS scenes. It combines pretrained navigation, HOI, and HSI experts through a structured scene representation and stage-wise task compilation. Object states are updated between stages to condition subsequent actions, while a boundary adapter improves motion continuity. An optional physics-based residual controller refines the generated motions. Evaluation includes comparisons with existing motion generators, component analyses, and a user study.

Critical strengths
  1. The paper addresses a relevant problem by composing pretrained navigation, HOI, and HSI experts into multi-stage sequences. Persistent object identities and scene-state updates support cross-stage dependencies, such as sitting on a relocated chair, without training an additional unified motion generator.

  2. The reported results show improved task completion over the evaluated baselines, and the boundary-adapter ablation provides quantitative evidence of reduced motion discontinuities.

Critical weaknesses
  1. The proposed harness is underspecified and insufficiently evaluated.

    • The claimed closed-loop orchestration is unclear. The paper never specifies which LLM generates the Activity Script. Although scene-state feedback is described, such as updating a chair's coordinates, feedback to the high-level planner is not established. Without evidence that execution outcomes affect planning decisions, the claimed loop may amount to state updates within a predefined workflow, making any implication of adaptive planning/harness design misleading.

    • Agent-harness baselines are missing. If the authors claim a contribution at the agent-harness level, comparison with strong general-purpose agent baselines is essential, e.g., GPT-6-astra with Codex and Claude Opus 5.5 with Claude Code. These baselines should receive the same motion experts, scene-state interfaces, and task information under comparable execution budgets.

  2. The physical-simulation module is inadequately specified, presented, and evaluated.

    • The human representation is unclear, and the demos are misleading. The paper does not clearly specify its human representation. The demos consistently use a rigid-body humanoid commonly associated with physics-based character control, blurring the distinction between kinematic animations and simulated rollouts. Only two physics clips are provided in the supplementary, and Case 5 still exhibits obvious penetration with the scene.

    • The simulation evaluation is insufficient. Table 4 reports only reference-tracking errors on an unspecified number of short clips averaging about 10 seconds, with no baseline controller or residual-refinement ablation. Beyond qualitative examples, there is no before/after comparison of penetration, foot sliding, contact quality, or task success.

    • The module's applicability to the main results is unclear. Many supplementary kinematic demos exhibit obvious foot sliding but are not shown after physical refinement. It remains unclear whether refinement was simply omitted or whether poor reference-motion quality prevents successful tracking.

Questions for authors
  1. What motivates the use of 3DGS? Planning, motion generation, and physical execution appear to rely primarily on geometric proxies and structured scene state. What does 3DGS contribute beyond visualization, and how does this motivate the Interactive Scene Package (ISP) design? Recent human and humanoid works establish concrete roles for 3DGS [1-4]:

    • ZeroHSI (3DV 2026) uses 3DGS and differentiable rendering to optimize 3D HSI sequences
    • GenHSI (WACV 2026) uses 3DGS to render spatially consistent human-scene interaction keyframes that condition video generation
    • LEGS (CoRL 2026) uses 3DGS for photorealistic rendering to reduce the visual sim-to-real gap for humanoid robots
    • VLK (CoRL 2026) uses 3DGS to render egocentric observations paired with synthesized motion trajectories to build their training dataset
  2. How does the planner operate? Are the evaluated scripts automatically generated or manually specified/corrected? If a chair fails to reach its target, does the planner detect failure, retry, switch experts, or does the system only update numerical conditions for the existing script?

  3. What human representation is used, and which demos are simulated? Please specify the common body model, conversion across experts, and mapping to the displayed/simulated humanoid. Please clearly label kinematic versus simulated results and use conventional human-mesh visualizations for kinematic motion. Which scene geometry participates in collision handling, and what explains the penetration in Case 5?

  4. How broadly does physical refinement work? How many sequences were evaluated, and was refinement used for Table 1? Why were the supplementary sequences with obvious foot sliding not refined: was simulation omitted, or did tracking fail?

[1] Li et al. ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation. 3DV 2026.

[2] Li et al. GenHSI: Controllable Generation of Human-Scene Interaction Videos. WACV 2026.

[3] Kim et al. LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World. CoRL 2026.

[4] Wang et al. VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes. CoRL 2026.

Reviewer_AwdTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents World2HOSI, a multi-expert orchestration framework that synthesizes long-horizon 4D Human-Object-Scene Interaction (HOSI) sequences in reconstructed 3D Gaussian Splatting (3DGS) environments from natural language instructions. The core components include: (1) an Interactive Scene Package (ISP) that decouples visual 3DGS assets from geometric proxies and dynamically updates object poses, local support surfaces, and navigation occupancy grids across stages; (2) a Task Compiler that grounds LLM-generated semantic scripts into stage-wise numerical conditions (MotionIR); (3) a directional Cross-Expert Reconciliation adapter using quintic smoothstep projection to stitch adjacent motion segments; and (4) an optional RL-based bounded residual controller to refine physical plausibility. Quantitative experiments on 60 sequences across 10 scenes (20 task definitions), diversity evaluations on 90 sequences, and a 30-participant perceptual user study validate that World2HOSI outperforms InfBaGel, HOSIG, and single-expert baselines in task completion and geometric plausibility.

Critical strengths

1. Stateful Closed-Loop World Formulation

A central bottleneck in prior HSI and HOI literature is that scene conditioning is treated as static, making multi-stage tasks with cross-stage object dependencies (e.g., relocating a chair and subsequently sitting on its updated pose, or moving an obstacle out of a navigation path) intractable for single experts.

2. Causal Cross-Expert Boundary Reconciliation

Unlike bidirectional motion blending which corrupts the terminal contact configuration achieved at the end of an interaction stage, the proposed forward-directional projection adapter (Eq. 3–6) strictly preserves the preceding terminal state while smoothly decaying the root and rotational offset over the incoming prefix. This simple yet effective design ensures that achieved object placements remain anchored while reducing boundary velocity jumps by 67.3% over bidirectional blending (Table 3).

Critical weaknesses

1. Unsubstantiated Efficacy of the Physics Refinement Module

The role and actual contribution of the RL-based bounded residual physics module (Section 3.4 and Appendix A.4) are poorly substantiated in the experimental validation. First, Figure 2 marks physical simulation as "Optional", and Appendix A.5 explicitly states that the user study only evaluated "kinematic videos". Measuring tracking error alone is fundamentally flawed for validating physics refinement, as an identity policy that does nothing achieves zero tracking error.

2. Unverified "Data Synthesis" Utility

While the paper repeatedly positions World2HOSI as a data synthesis engine to resolve the training data bottleneck for unified HOSI models (analogous to BEDLAM or InHabit in Section 1), the evaluation is limited to 60 benchmark sequences and 90 diversity clips, without any downstream experiment showing that training or fine-tuning a unified generator on World2HOSI-synthesized data improves performance.

Questions for authors

1. PPO Generalization and Computational Overhead

In Appendix A.4, is a single universal residual policy trained across all clips, or is PPO optimized per sequence at test time to track each kinematic reference? Please report the wall-clock runtime breakdown per 10-second sequence.

2. Degree of Automation in ISP and Activity Scripts

Appendix A.1 mentions manual inspection and correction for ISP assets, and Section 3.2 notes that Activity Scripts can be generated by an LLM or specified manually. For the 60 benchmark sequences (Table 1), were the Activity Scripts generated autonomously by the LLM from raw text instructions, or manually authored?

3. Clarification and Ablation on Physics Refinement (Tables 1 and 4)

Are the numbers for "Ours" in Table 1 computed on purely kinematic outputs (matching the user study in Appendix A.5) or after PPO physics refinement? Please provide an explicit side-by-side comparison of kinematic outputs vs. PPO-refined outputs across all physical plausibility metrics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.