acceptodds
Under review as a conference paper at ICLR 2027

PriorHOSI: Composing Motion Priors for Human–Object–Scene Interaction Generation

Abstract

Human–object–scene interaction (HOSI) generation aims to synthesize coordinated human and object motion compatible with the surrounding scene. Existing HOSI datasets remain scarce, while high-quality, diverse human–object interaction (HOI) and human–scene interaction (HSI) datasets offer complementary supervision. The challenge is to adapt human–object motion to scene geometry while preserving the contact relationships that coordinate the interaction. We propose PriorHOSI, a framework that learns a human–object interaction prior (HOIPrior) and a human–scene interaction prior (HSIPrior) separately and composes them at inference time. The key idea is to use HSIPrior’s scene-adapted human motion to guide joint updates of the body and object through HOIPrior. Diffusion noise optimization realizes these updates by fitting the generated motion to the scene-adapted target while constraining hand positions in the object’s coordinate frame. To support repeated updates, HOIPrior uses relation-aware consistency distillation to reduce sampling steps and retain hand–object geometry. To better exploit scene geometry, HSIPrior uses body-part-aware residual refinement to propose scene-aware motion adjustments during editing. Experiments on the InfBaGel benchmark demonstrate that PriorHOSI achieves strong task performance and hand–object contact while improving human–scene compatibility.

Primary area:
generative models
Keywords:
Human-Object-Scene Interaction, Motion Generation, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_oTxKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PriorHOSI, which separately learns a human–object interaction prior from OMOMO and a human–scene interaction prior from LINGO, then composes them at inference time. The HOI prior uses relation-aware consistency distillation, while the HSI prior uses body-part-aware SDF residual refinement. During editing, diffusion noise optimization moves the joint human–object motion toward refreshed scene-adapted human targets, while an augmented-Lagrangian constraint preserves object-relative hand positions. On 469 InfBaGel tasks, the paper reports higher task success and hand–object contact, and lower body–object and human–scene penetration than InfBaGel. Object–scene penetration is mixed: the mean and maximum depths increase, while the fraction of penetrating frames decreases.

Critical strengths
  1. A useful composition strategy for complementary interaction data. Separately learning HOI and HSI priors addresses the mismatch between coupled object motion and scene-conditioned human motion. Constraining hand positions in the object frame provides a principled way to preserve local contact while allowing global body–object adjustments.

  2. A substantial benchmark evaluation with informative ablations. The paper evaluates on 469 InfBaGel tasks, compares against several methods, and includes ablations for scene guidance, contact constraints, relation supervision, and residual refinement. It also reports the object–scene trade-off rather than presenting every metric as an improvement.

Critical weaknesses
  1. The optimization objective used for the reported results is not traceable from the paper to the supplementary implementation. Section 3.4 defines the editing objective as scene-target matching plus the contact constraint. However, the default supplementary protocol appears to activate a MetricMotionObjective, which the closed-loop implementation adds to the HSI and relation losses. The objective includes terms for goals, body–object collision, human–scene and object–scene penetration or clearance, foot stance and speed, motion continuity, and trajectory acceleration. Several of these directly target reported evaluation metrics. If these terms were active for Tables 1–3, the method description and ablations omit consequential parts of the optimization. If the supplement represents a different or later configuration, the paper should identify which code and configuration produced the reported results.

  2. The benchmark development history needs clarification. The supplementary summary code marks test_set_development as True and states that all 469 tasks had historical development exposure. This flag alone does not establish that test outcomes informed method or hyperparameter choices, but the paper does not explain what was exposed or what decisions followed. Separately, the reported object–scene mean and maximum penetration worsen, and the task-success gain over InfBaGel is 1.71 percentage points without uncertainty estimates in the paper.

Questions for authors

Q1: Were the additional metric-aware terms in the default supplementary protocol active? If so, please report their contribution and ablate them against the objective stated in Section 3.4. If not, please identify the configuration that reproduces the tables.

Q2: Were the additional metric-aware losses applied to the baselines, or are they part of the benchmark’s prescribed evaluation protocol? If they were applied only to PriorHOSI, the comparison may conflate prior composition with direct optimization of the evaluation metrics. If they are part of the protocol, please clarify where it specifies them and how they are used for each method.

Q3: What does the supplement’s test_set_development: True flag mean in this submission? Were method choices, hyperparameters, or stopping decisions made after examining outcomes on these tasks? Please clarify the development history and, if those outcomes informed the final choices, provide an evaluation on tasks not used during development.

Q4: Could you report paired uncertainty estimates for task success and the scene-penetration metrics, and discuss the increase in object–scene mean and maximum penetration?

Reviewer_qGrCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses HOSI generation by training two specialized priors—an object-coupled HOIPrior from OMOMO and a scene-conditioned HSIPrior from LINGO—and composing them at inference via diffusion noise optimization. Object-frame hand constraints preserve contact, while a scene-adapted human target guides joint body–object updates, yielding higher contact rates and lower penetration on the InfBaGel benchmark.

Critical strengths
  1. The motivation is clear: HOSI data are scarce, and forcing one model to handle both human–object and human–scene aspects creates conflicting objectives. Training two separate priors and composing them is a simple answer to this.

  2. The results are good. In particular, when the two priors are combined, both contact and penetration improve at the same time, which supports the main claim.

Critical weaknesses

The overall quality of this paper is good. There are minor issues:

  1. In line 203, there seems a typo when you describe the elements of c_{O}, it should be G_{O}.

  2. The 16 sampling steps with 150 DNO iterations seems a lot compute. A runtime comparison with other methods, as well as results on the choice of sampling steps and DNO iterations, would help assess the cost.

Questions for authors
  1. What was the number of samples generated per task, how were they selected, and were the reported results averaged across runs?

  2. Under identical HOIPrior, initialization, contact constraints, and optimization budget, how does PriorHOSI perform when the HSI target is replaced with direct SDF-based collision guidance?

Reviewer_BSoFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PriorHOSI, a framework for human–object–scene interaction (HOSI) generation that separately trains human–object interaction (HOI) and human–scene interaction (HSI) motion priors and composes them at inference time.

Specifically, HOIPrior is trained on OMOMO to learn coordinated human–object motion, while HSIPrior is trained on LINGO to learn scene-aware human motion. The two priors are integrated through diffusion noise optimization (DNO), which optimizes the initial noise of HOIPrior using scene-adapted motion targets from HSIPrior and object-relative contact constraints to preserve human–object coordination.

The authors further introduce relation-aware consistency distillation to improve HOIPrior's sampling efficiency and contact preservation, along with body-part-aware residual refinement to enhance HSIPrior's scene adaptation.

Experiments on the InfBaGel benchmark compare PriorHOSI against TRUMANS, LINGO, CHOIS, HOSIG, and InfBaGel. PriorHOSI achieves the highest task success rate (83.16%) and human–object contact rate (87.54%), as well as the lowest body–object penetration (3.17), foot sliding (0.10), and human–scene penetration across all three reported metrics.

Critical strengths
  1. Effective framework for composing specialized HOI and HSI priors. The paper proposes a well-motivated framework that separately learns human–object interaction (HOI) and human–scene interaction (HSI) priors and composes them through inference-time optimization. This design provides a good alternative to unified training, allowing each prior to specialize in its respective interaction domain.
  2. Effective utilization of complementary interaction datasets. The proposed framework makes good use of the complementary supervision from OMOMO and LINGO without requiring large-scale paired HOSI data. The inference-time composition provides an effective way to integrate human–object coordination and scene-aware motion generation while addressing their potentially conflicting objectives.
  3. Well-designed optimization objectives and strong empirical results. The combination of scene guidance and object-relative contact constraints is intuitive and effective. The proposed relation-aware consistency distillation and body-part-aware residual refinement further improve the two specialized priors. Experimental results demonstrate strong performance compared with existing baselines, particularly in human–object contact quality, task success, and human–scene compatibility. The ablation studies also provide evidence for the complementary roles of scene guidance and contact preservation.
Critical weaknesses
  1. Insufficient controlled comparison to establish the advantage of separate-prior composition over unified training. Although PriorHOSI outperforms InfBaGel, the current comparison does not fully isolate the benefits of separately trained HOI and HSI priors from those of additional inference-time optimization. In particular, PriorHOSI performs 150 iterations of diffusion noise optimization (DNO), whereas InfBaGel is evaluated using its original inference pipeline. Since InfBaGel already adopts a unified framework for learning human–object–scene interactions, a more informative baseline would be to apply DNO to the pretrained InfBaGel model while retaining its existing inference pipeline. Comparing InfBaGel + DNO with PriorHOSI under comparable test-time computational budgets would help determine whether the improvements primarily stem from separate-prior composition or from additional test-time optimization.
  2. Insufficient analysis of DNO iterations and inference efficiency. The proposed method uses 150 DNO iterations, each requiring backpropagation through a 16-step HOIPrior sampler. However, the paper does not sufficiently justify the choice of 150 iterations or analyze the trade-off between optimization steps, generation quality, and computational cost. An ablation over different numbers of DNO iterations (e.g., 25, 50, 100, and 150), together with end-to-end inference latency and memory usage, would help clarify the convergence behavior and practical efficiency of the proposed framework.
Questions for authors

Could the authors evaluate InfBaGel + DNO to provide a fairer comparison with PriorHOSI?

A possible approach is to freeze the pretrained InfBaGel model, retain its original dynamic perception and sampling guidance, and apply DNO to optimize the initial noise using scene collision and contact preservation losses. The scene information could be refreshed after each iteration, with a computational budget comparable to PriorHOSI.

Could the authors discuss the feasibility of this experiment and, if possible, provide results? This comparison would help assess how much of the performance gap can be reduced by additional test-time optimization, although it would not fully isolate the contribution of the separate-prior design.

Reviewer_EZ9fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PriorHOSI separately trains a human–object prior on OMOMO and a human–scene prior on LINGO, then composes them at inference time through diffusion noise optimization. Scene-adapted human targets guide joint human–object updates, while object-relative hand constraints preserve contact geometry. The paper reports improved contact and human–scene metrics over hybrid InfBaGel on the 469-task benchmark, with mixed object–scene penetration results.

Critical strengths
  1. The paper addresses an important HOSI generation problem and makes use of complementary HOI and HSI supervision.

  2. The reported results show improved contact and human–scene metrics over the hybrid InfBaGel baseline.

Critical weaknesses
  1. The contribution and motivation of the fusion strategy are relatively limited and insufficiently established. The paper claims to address the problem of fusing HOIPrior and HSIPrior, but it does not discuss the limitations of directly combining the two priors or sufficiently review relevant prior work. Consequently, the motivation for the proposed fusion strategy is not clearly explained.

  2. The relationship between the prior-specific improvements and the central fusion problem is unclear. The modifications to HOIPrior and HSIPrior appear to address issues specific to the two priors independently. As presented, the method appears to be a combination of incremental improvements to the two priors, rather than a solution derived from the core fusion problem.

Questions for authors
  1. What are the specific limitations of directly combining HOIPrior and HSIPrior, and how does the proposed fusion strategy differ from the closest prior work on motion-prior composition?

  2. How does each modification to HOIPrior and HSIPrior address the central fusion problem, rather than only improving the corresponding prior independently?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.