acceptodds
Under review as a conference paper at ICLR 2027

HOHI4D: Scalable Synthesis of 3D Human-Object-Human Interaction Data

Abstract

3D human-object-human interaction (HOHI) modeling remains underexplored due to the lack of scalable datasets and the difficulty of acquiring accurate multi-agent interaction data. Existing approaches rely on controlled capture setups or strong assumptions on object pose and contact, limiting their applicability in complex scenarios. In this work, we propose a fully automated and controllable pipeline for large-scale HOHI data synthesis, combining prompt-driven video generation with SAM3D-based reconstruction. Our framework enables diverse interaction generation, reduces object drift via geometry-aware refinement, and removes reliance on predefined object templates. Based on this pipeline, we construct HOHI4D, a large-scale dataset with over 5,000 sequences and 13 hours of motion. Furthermore, we introduce a structured generative model for HOHI. We design HOHI-VAE to learn disentangled latent representations of individual motion, object dynamics, and global interaction, and build HOHI-Diff, a latent diffusion model operating in this structured space. Our approach enables stable and coherent generation of complex multi-agent interactions, providing a scalable benchmark for HOHI modeling.

Primary area:
generative models
Keywords:
3D Human Motion, Generative model, Human-Object Interaction, Human-Human Interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gW7cOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

HOHI4D provides approximately 5,000 synthetic sequences of two-person, shared-object interactions. Its reconstruction pipeline accompanies HOHI-VAE's structured representation and HOHI-Diff's text-and-object-conditioned generation, with reported improvements over adapted MDM, MLD, and CHOIS baselines.

Critical strengths
  1. The task of coordinated interaction between two people and a shared object is interesting and valuable.
  2. The scalable synthesis pipeline addresses an important interaction-data bottleneck, with refinement improving all three reported reconstruction measures.
  3. Component ablations, training details, prompt examples, and failure cases provide useful supporting evidence.
Critical weaknesses
  1. The reconstruction-derived dataset lacks reported independent 3D accuracy validation, leaving its reliability as ground truth uncertain given the apparent jitter and interpenetration in the videos.
  2. The evaluation relies on 2D overlap and learned embedding metrics without directly measuring 3D pose, trajectory, or contact accuracy, leaving geometric correctness insufficiently established.
Questions for authors

How are the dataset splits separated by object instance and prompt template, and which split is used to train HOHI-CLIP?

1 comment

Thank you for the clarification and additional evaluations, which help address my main concerns. Please include these results and clearly state the remaining limitations in the revised paper. I maintain my weak acceptance recommendation.

Reviewer_rzhYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a pipeline for synthesizing 3D human-object-human interaction data from generated videos. It uses structured prompts and image/video generation, reconstructs human motion and object geometry with SAM3D-based methods, and refines object poses through geometric alignment and temporal filtering. The authors introduce HOHI4D, with over 5,000 sequences and 13 hours of motion, and propose HOHI-VAE and HOHI-Diff, which model the two human streams, object motion, and global interaction in a structured latent space. They evaluate object reconstruction against CARI4D and SAM3DO, and compare generation with adapted MDM, MLD, and CHOIS using a proposed HOHI-CLIP evaluator.

Critical strengths
  1. The paper tackles a meaningful data bottleneck. Over 5,000 sequences and 13 hours could be useful for a setting that is less represented than single-human, human-object, or human-human motion. The automated generation and reconstruction pipeline is a potentially valuable contribution if the resulting interactions are validated.

  2. The structured model is a reasonable approach to the HOHI setting. Separating the two human streams, object dynamics, and a fused interaction representation is well motivated. On the paper’s proposed HOHI-CLIP metrics, HOHI-Diff improves over the adapted baselines on several reported measures, and the ablations suggest that the fusion encoder and object shape tokens affect performance.

Critical weaknesses
  1. The quality and breadth of the synthetic dataset are not independently established. The videos are generated and then reconstructed from monocular input. The prompt design imposes substantial constraints: clean indoor scenes, a static camera, full-body visibility, minimized occlusion, and scripted interaction patterns (Appendix D). The paper acknowledges failures involving object deformation and human disappearance (Appendix H), as well as reconstruction artifacts and limited explicit physical modeling (Appendix I). It does not report a human audit of semantic interaction quality. Table 3 includes a “Ground Truth” row, but its source and relationship to HOHI4D or real-world data are not specified. If it refers to held-out HOHI4D sequences generated and reconstructed by the same pipeline, it provides an in-distribution reference but does not independently validate the fidelity of the synthesized 3D interactions against real data. If it comes from real recordings, the dataset and its role in model selection should be identified. This matters because HOHI4D is a central contribution and is also used in the generation experiments; scale alone does not establish reliable contact, object motion, or interaction semantics.

  2. The object-tracking metrics overlap with the refinement objectives, and their evaluation protocol is underspecified. The refinement procedure uses bounding-box and mask-based projection terms, while Table 2 reports bounding-box and mask mIoU as evidence of improvement. The paper does not specify the reference annotations, exact test subset, or collision-volume calculation, and it reports no independent 3D object-pose accuracy. It is therefore unclear whether the gains demonstrate better 3D tracking or mainly better fit to the 2D projection criteria used by refinement. Separately, the paper does not describe the HOHI4D train/validation/test split or provide enough detail about how HOHI-CLIP is trained and evaluated to determine whether the generation metrics are held out from training and model selection.

Questions for authors

Q1: What reference labels and exact test subset are used for the bounding-box and mask mIoU, and how is collision volume defined? Were these data used to tune the refinement weights? Is an independent 3D pose or contact measure available, given that refinement itself uses box- and mask-based projection terms?

Q2: What is the source of the “Ground Truth” row in Table 3, and how does it relate to the HOHI4D train/validation/test partitions or to real-world data? How is HOHI-CLIP trained and evaluated relative to those partitions? These details could materially affect the interpretation of Table 3. A limited human audit of a held-out subset or evaluation on clearly identified real HOHI data would further support the dataset-quality and generalization claims.

Reviewer_GgMwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the human-object-human interaction generation task. To obtain the paired human and object data, the paper first tries to use a video generation to generate the interaction videos and then use SAM3D models to reconstruct the human and object motion. In this way, the paper proposes the HOHI4D dataset and then based on this dataset, the paper further proposes the corresponding HOHI-VAE and HOHI-Diff model for the interaction generation.

Critical strengths
  1. The paper proposes a large human-object-human motion interaction dataset.
  2. Due to the lack of a large MOCAP human-object-human dataset, the automatic data collection framework with video generation model and SAM-3D models is well-motivated and well described.
  3. The presentation and the figures are generally clear.
Critical weaknesses
  1. I cannot find the AI use statement in this paper, which is necessary this year.
  2. Data quality looks bad in the supplementary video. Both human and object are shaking all the time in the provided demo.
  3. Although Table 2 provides some comparison of object reconstruction quality, it's not clear which dataset to evaluate and not enough to demonstrate the effectiveness of this data reconstruction pipeline. As it is a pipeline for human-object interaction data reconstruction, it’s important to provide both the human and object quality comparison on standard benchmarks such as the BEHAVE dataset (as in CARI4D) or Core4D dataset.
  4. The novelty of the model design is limited. The proposed model basically adopts DHVAE (ICLR 2026) from human-human interaction to human-object-human interaction by adding the corresponding object encoder-decoder. No other specific model or loss design for object interaction.
  5. There are no contact and penetration metrics in Table 3. As the task is about interaction generation, the author should report standard contact and penetration metrics as in CHOIS paper.
  6. In paragraph “Human-Object Motion Dynamics” in section E, the authors claim “Human-Object Motion Dynamics. We further examine the motion statistics of both humans and objects.”, but I cannot find these statistics.
Questions for authors
  1. As there are noticeable temporal jitter and contacts problems in the supplementary video, please provide a quantitative analysis on those metrics for the generated dataset and compare with commonly used dataset such as Core4D and BEHAVE dataset.
  2. Further explain Table 2 and provide full evaluation on both human, object and interaction evaluation for the proposed data collection pipeline with existing methods such as CARI4D on BEHAVE or Core4D dataset.
  3. Please add commonly used contact and penetration metrics in Table 3 and could potentially add some physics metrics (sliding, floating and jerk).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.