- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This submission proposes an agentic harness, named Word2HOSI to generate human-object-scene interactions over long time horizons from a given 3D scene and a text prompt as input. It first introduces a 3D scene representation (Interactive Scene Package) and then uses an LLM to plan and compile activity scripts, and then coordinates experts to generate interaction motions. It also incorporates a physics-based refinement for interactions. Experiments on 10 scenes and 20 tasks show improvement over baseline approaches.
Critical strengths
-
Many of the generated human interactions over long time horizons, provided in the supplemental, seem reasonable.
-
The evaluation results (Table 1) show improved performance of task completion, interaction plausibility, and human motion quality, when compared to strong baselines like HOI-FHLI. The perceptual study (Figure 6) provides further support in terms of visual realism and physical plausibility.
Critical weaknesses
-
The main concern is the unclear technical innovations. The authors claim to design a multi-expert harness, but the scene representation (Section 3.1) and task planning (Section 3.2) seem straightforward to me. Cross-expert reconciliation (Section 3.3) also seems trivial, though improving the motion transition. I don't feel like I have learned a lot after reading this work.
-
With the increasing capabilities of frontier models like GPT-Astra, such a harness design is questionable. I am curious how well GPT-Astra works to generate the human interactions without this harness.
-
In the experiments, there is no robustness test of this harness against different expert alternatives (Section A.2.2). In fact, the proposed harness uses HOI-FHLI for dynamic object interaction generations, but it's unclear why it achieves better object motion performance in Table 1.
-
The interaction generation results shown in the supplemental still have noticeable issues like foot sliding and physical penetration (like the book stack handling). It is questionable whether the physics-based refinement is effective or not.
-
The ablation studies did not include justifications for the contributions listed in Section 1, including the structured scene-state representation and the closed-loop generation mechanism. For the ablation of bounded residual refinement, there is no corresponding elaboration in the methodology section.
-
The long horizon generations lack quantitative results.
-
Related Work (Section 2) lacks discussions of relevant works on harness designs for 3D/4D generation (although not directly on human interaction generation) and how this harness design differs from them at the conceptual level.
-
The writing needs improvement. The current version reads like LLM-generation heavy across sections, and several sections are hard to decipher, including Section 3.2 - Compilation and Section 3.4.
Questions for authors
My questions are mainly on the technical innovation and differentiation, unclear performance against frontier models, unclear gain over HOI-FHLI, and missing ablations, as outlined above in the Critical Weaknesses.