acceptodds
Withdrawn from ICLR 2027

OmniHOI: Generalizable Human-Object Interaction Generation through Explicit Interaction Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Synthesizing human-object interaction (HOI) is valuable for applications in robot imitation learning and AR/VR animation. However, existing methods are ineffective in handling multiple-object interactions, or otherwise specialized to particular object configurations. Meanwhile, they fail to generalize to object categories or trajectories unseen during training. We present OmniHOI, a single HOI framework that jointly tackles single-object, articulated-object and multiple-object interactions. It devises a hierarchical generation pipeline, consecutively outputting object motion from text instructions, hand motion from object motion, and full-body motion from both object and hand motion. To reinforce interaction information, OmniHOI introduces interaction-aware self-conditioning and interaction-informed loss for diffusion models. OmniHOI also introduces multi-object test-time optimization to enhance hand contacts with multiple objects. OmniHOI sets a new state of the art on GRAB, ARCTIC and HIMO dataset, achieving stronger motion quality, text alignment, and hand–object interaction accuracy. Conditioned on unseen object categories or object trajectories, OmniHOI demonstrates stronger out-of-distribution generalization than prior work. Figure 1 summarizes our findings.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Human-Object Interaction, Motion Generation, Diffusion Models, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PH3DOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary
  1. The paper is generally well written and easy to follow. The hierarchical decomposition of object motion, hand motion, and full-body motion gives the overall framework a clear structure.

  2. The experimental coverage is relatively comprehensive. The authors evaluate single-object, articulated-object, and multi-object interaction generation using GRAB, ARCTIC, and HIMO, and the method achieves strong quantitative results across these settings.

  3. The paper pays explicit attention to interaction modeling rather than relying only on generic motion-generation objectives. The interaction-aware self-conditioning and interaction-informed loss are conceptually reasonable ways of introducing object-object and hand-object geometric information into diffusion models.

  4. The multi-object setting is meaningful and relatively challenging. Predicting which hand interacts with which object and refining the corresponding contact provides a practical mechanism for handling interactions involving multiple objects.

  5. The paper also considers out-of-distribution settings, including unseen object trajectories and unseen object categories, which is useful for studying generalization beyond standard in-distribution HOI generation.

  6. Overall, the implementation appears technically solid, and the reported results suggest that the proposed combination of components is effective in practice.

Critical strengths
  1. The paper is generally well written and easy to follow. The hierarchical decomposition of object motion, hand motion, and full-body motion gives the overall framework a clear structure.

  2. The experimental coverage is relatively comprehensive. The authors evaluate single-object, articulated-object, and multi-object interaction generation using GRAB, ARCTIC, and HIMO, and the method achieves strong quantitative results across these settings.

  3. The paper pays explicit attention to interaction modeling rather than relying only on generic motion-generation objectives. The interaction-aware self-conditioning and interaction-informed loss are conceptually reasonable ways of introducing object-object and hand-object geometric information into diffusion models.

  4. The paper also considers out-of-distribution settings, including unseen object trajectories and unseen object categories, which is useful for studying generalization beyond standard in-distribution HOI generation.

  5. Overall, the implementation appears technically solid, and the reported results suggest that the proposed combination of components is effective in practice.

Critical weaknesses
  1. Limited methodological novelty. The overall technical contribution appears somewhat incremental. The hierarchical decomposition itself follows prior work, while self-conditioning, geometric interaction representations, diffusion-based generation, and test-time optimization are also established techniques. The main novelty is largely in combining these components and injecting interaction information into different stages of the pipeline. The paper would benefit from a clearer explanation of what fundamentally new modeling insight is introduced beyond this integration. In particular, the current ablation does not compare the proposed interaction-aware self-conditioning against standard self-conditioning, making it difficult to determine whether the gains come from the proposed interaction representation or simply from adding self-conditioning.

  2. Limited diversity of full-body interaction dynamics. Although the paper claims generalizable full-body HOI generation, the evaluated interactions are predominantly localized object manipulation. There is relatively limited evidence for interactions involving substantial lower-body movement, long-range root translation, walking while carrying objects, approaching or leaving objects, or dynamic pushing/pulling behaviors. Therefore, the demonstrated generality is mainly over object configurations rather than over full-body interaction dynamics. Evaluation on more locomotion-rich HOI data, such as OMOMO and BEHAVE-style interaction scenarios, would substantially strengthen the claim.

  3. The "unified" claim is somewhat overstated. The paper repeatedly emphasizes that OmniHOI provides a single framework across different interaction settings. However, the authors actually train one model per benchmark, rather than jointly training a single model across GRAB, ARCTIC, and HIMO. Therefore, the current results mainly demonstrate a unified architecture rather than a truly unified HOI model. This distinction should be made clearer, or joint cross-dataset training should be investigated.

  4. The OOD trajectory experiment is not an end-to-end evaluation of trajectory generalization. For this experiment, the ground-truth object trajectories from the held-out test set directly replace the output of the object-motion generation stage. Therefore, this experiment evaluates whether the hand and body generators can follow externally provided unseen trajectories, rather than whether the complete OmniHOI framework can generate unseen object trajectories from text. The corresponding generalization claim should be stated more carefully.

  5. Error propagation in the hierarchical pipeline is insufficiently studied. The object-motion, hand-motion, and full-body generators are trained separately, so errors from early stages can naturally propagate to downstream stages. However, the paper does not provide oracle-style experiments to quantify this effect. For example, comparisons using GT object motion versus generated object motion, or GT hand trajectories versus generated hand trajectories, would help reveal where the major errors originate and how robust the later stages are to imperfect intermediate predictions.

  6. The physical plausibility analysis is not fully convincing. While OmniHOI improves several contact-related metrics, it does not consistently improve penetration. This suggests a possible trade-off between increasing contact and introducing excessive interpenetration. The paper should discuss this issue more carefully and ideally provide additional analysis showing whether improved contact accuracy actually corresponds to physically valid interaction.

Questions for authors

I would like to ask the author if they checked the performance on the OMOMO dataset, which involves more complex full-body locomotion and manipulations. For the other questions, please refer to the weaknesses.

Reviewer_17z8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OmniHOI, which generates human-object interactions by a hierarchical pipeline that generating object motion, hand motion, and full-body motions step by step by diffusion models. Its main contribution lies in object–object geometric self-conditioning, a hand-object interaction loss, and learned hand-object assignment for contact refinement. The same architecture, trained separately per dataset, can generate rigid, articulated, and multi-object interactions. Validation on GRAB, ARCTIC, and HIMO show the method has merit and can support generalization to HUMOTO from HIMO training.

Critical strengths

While most existing methods focus on interactions with a single rigid object, this paper presents one of the few frameworks that supports both articulated-object and multi-object interactions. Object masking allows the same architecture to accommodate different numbers of objects.

The paper shows consistent benchmark improvements, thought the baselines looks weak by the adapation

Critical weaknesses

I feel the major self-conditioning mechanism is insufficiently specified.

  • Dex2HOI already incorporates hand-relative object representations and contact-aware supervision, making the proposed geometric feedback across denoising steps a major methodological distinction.
  • However, Section 3.3 primarily defines the feedback representation without explaining how self-conditioning inputs are constructed during training. It remains unclear whether an auxiliary prediction is used, how gradients through the feedback are handled
  • Appendix A does not resolve these details. These choices affect training-inference consistency and reproducibility, and how this is handled if there is a gap between training and inference
  • There is also no comparison with standard self-conditioning or corresponding ablation of generation quality. Figure 3 compares complete systems, while the supplementary video does not isolate the mechanism’s qualitative effect.
  • Some important designs such as stage-wise generation (e.g., CHOIS) and post-generation optimization (e.g., SAGA and InterDreamer) have already been explored in prior work, leaving the self-conditioning important to be well explained

The qualitative examples are too limited to show whether the proposed design choices consistently improve human-object interaction. More visualizations and videos, including randomly selected examples and failure cases, would help assess how reliable these improvements are and reduce concerns about cherry-picking.

Questions for authors

One intuitive design is to have direct motion self-conditioning by feeding the model’s previous estimate of clean motion directly back into the denoiser, compared to the proposed geometric self-conditioning in this paper.

Will geometric self-conditioning affect the generalization?

Reviewer_WCGkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets text-conditioned human-object interaction (HOI) generation that covers single-object, articulated-object, and multi-object settings with one architecture and generalizes to unseen object trajectories and categories.

OmniHOI follows a hierarchical object →\rightarrow hand →\rightarrow full-body diffusion pipeline and adds three interaction-centric components: interaction-aware self-conditioning (relative-pose and BPS features from the previous estimate fed back as conditioning), an interaction-informed hand-object loss, and multi-object test-time optimization driven by a learned target-object/engaged-hand detector.

On GRAB, ARCTIC, and HIMO (2/3 objects), plus GRAB with ground-truth object trajectories and cross-dataset HUMOTO, it reports the best FID, R@3, and HOI-ADD among the compared methods, with a component ablation on HIMO 2-object and a runtime of 3.77 s per ~310-frame sequence.

Critical strengths
  1. One architecture with consistent gains across very different HOI settings. The same pipeline, trained per dataset, achieves the best FID, R@3, and HOI-ADD on single-object (GRAB), articulated (ARCTIC), and 2-/3-object (HIMO) benchmarks (Tables 2--3), whereas the strongest baseline changes from setting to setting. Prior methods are each specialized to a subset of these configurations (Table 1), so a single framework that is competitive everywhere is the paper's most consequential contribution.

  2. Explicit interaction interfaces that are practical and reusable. Making object motion an explicit intermediate lets ground-truth or external trajectories replace stage one without retraining (Sec. 4.3), and the lightweight target-object/engaged-hand detector resolves the target-ambiguity problem of BPS-based contact optimization in multi-object scenes (Sec. 3.5, Appx. B.1), and the detector plus hand--object optimization together add only 0.13 s per sequence (Appx. D). This is a sensible and efficient design compared with noise-optimization methods that back-propagate through the whole sampling chain.

Critical weaknesses
  1. The headline claims rest on learned-embedding metrics whose reliability is not established, and physical plausibility is not better. FID and R@3 are computed in a TMR-style embedding space (Appx. C.3) whose evaluator training is not described (a dataset-specific evaluator is mentioned only for HUMOTO, Appx. C.1), yet several numbers are hard to interpret: OmniHOI's R@3 on GRAB (56.15%) exceeds that of real data (45.49%), real data reaches only 14.47% R@3 on HUMOTO, and the ARCTIC FID gap (2.26 vs. 16.63) is unusually large. No standard deviations or multiple seeds are reported for any table. Meanwhile, the physical-interaction metrics do not support "places the hands most accurately": penetration is higher than the real data and most baselines on ARCTIC (8.44 mm vs. 2.87 mm real) and GRAB (7.19 vs. 4.76 mm), and contact ratio is not the best on ARCTIC (9.52% vs. CoDA 10.15%). OmniHOI's interaction advantage rests mainly on HOI-ADD/OO-ADD, which use a coarse AUC up to 15 cm, while the finer contact and penetration measures are mixed. This matters because the paper's main claim is state-of-the-art realism and interaction accuracy.

    • Suggestion: describe evaluator training, report a reference FID computed between two disjoint subsets of the real test motions (the "Real" rows currently leave FID blank), report mean ±\pm std over seeds, and discuss the penetration results (e.g., before/after test-time optimization).
  2. Incremental technical novelty, with an ablation that does not isolate the new components. The hierarchical object →\rightarrow hand →\rightarrow body decomposition and the BPS-distance-based hand-object test-time optimization follow CoDA (Pi et al., 2025), as the paper states (Sec. 3.2, 3.5, Appx. B.2); the new parts are the feature-level self-conditioning, the interaction loss, and the target-object detector. The ablation (Table 5) is limited to HIMO 2-object, reports only ADD metrics, leaves four of the 16 entries blank, does not define the "w/o Hierarchical Generation" variant, and does not ablate the detector or test-time optimization. CoDA, the closest hierarchical baseline, is also not evaluated on HIMO (Table 3). In addition, the "OOD trajectory" setting uses ground-truth object trajectories from the standard GRAB test split (Appx. C.1; the Real row equals that of Table 2), so it is better described as trajectory-conditioned generation than as out-of-distribution generalization. Without these, it is unclear how much of the gain comes from the proposed components rather than the inherited pipeline.

    • Suggestion: isolate the self-conditioning, interaction loss, target-object detector, and test-time optimization in the ablation, define the "w/o Hierarchical Generation" variant, evaluate CoDA on HIMO, and describe the GRAB setting as trajectory-conditioned generation.
Questions for authors

Clarifications (could change my assessment):

  1. How were the text-motion evaluators for FID/R@3 on each dataset trained (data, architecture, checkpoint selection)? Why does OmniHOI's R@3 on GRAB (56.15%) exceed that of real motions (45.49%)? Are the reported numbers from a single run? What are the standard deviations across seeds/samples for the main metrics in Tables 2--4?
  2. Why is penetration on ARCTIC and GRAB higher than both the real data and most baselines, and how does it change with and without the test-time optimization?
  3. What does "w/o Hierarchical Generation" denote in Table 5, and why are HOI-ADD/OO-ADD missing for two of the variants? What is the effect of removing the target-object detector and the hand-object test-time optimization?
  4. Is the "OOD trajectory" set (Table 4, top) the same subject-s10 GRAB test split used in Table 2, conditioned on its ground-truth object trajectories? If so, in what sense are these trajectories out of distribution?

I am open to revisiting my rating if the rebuttal addresses these points well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.