acceptodds
Under review as a conference paper at ICLR 2027

InterAssembly: Assembling Long-Horizon Human-Object Interactions via Retrieval, Optimization, and Generation

Abstract

Human-object interaction (HOI) datasets have advanced motion synthesis and embodied behavior modeling, but remain limited to short, isolated clips. This clip-level supervision cannot capture long-horizon activities that require temporally coherent manipulation of multiple objects and consistent scene evolution, essential for embodied intelligence. Collecting such data is prohibitively expensive, and no existing method generates long-horizon, multi-object HOI activities under today's data budgets. We introduce InterAssembly, an agentic system that assembles activity-oriented long-horizon HOI animations from an HOI database of short captured clips. Given only a 3D scene and this HOI database, an LLM agent iteratively proposes the next sub-action, retrieves candidate reference clips by structural scene similarity and selects among them by semantic fit to the proposed action, and adapts object placements to the target scene using collision-aware feedback. An interaction mesh retargeting stage then transfers the retrieved human motion to the new scene while preserving bone lengths, foot grounding, and temporal smoothness. The retargeted joint trajectories serve as sparse control signals for a controllable masked motion generation model, which synthesizes dense, smooth full-body motion consistent with the established interaction structure. InterAssembly is, to our knowledge, the first system to generate coherent, long-horizon, multi-object HOI activities, including tool use, object transfer, and multi-step tabletop cooking, at durations far beyond existing HOI generation methods, with low human-object penetration and consistent scene evolution throughout.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-object Interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_a8nEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This submission introduces InterAssembly, a training-free approach for long-horizon human-object interaction (HOI) generation. Unlike methods that typically rely on training generative models, InterAssembly assembles coherent, long-horizon, multi-object activities, which do not exist in the original dataset, by leveraging a library of short-duration human motion capture clips. This process is structured into four decoupled stages: LLM-based planning, retrieval, retargeting, and synthesis. The method was evaluated on 15 self-generated long-horizon sequences using both physical/kinematic metrics and a user study.

Critical strengths
  • A new paradigm for long-horizon human-object interaction (HOI) generation is proposed. Unlike other approaches that rely on training generative models, this method can generate long-horizon tasks without requiring the training of any new models.

  • The proposed method is modular and training-free. The four stages, planning, retrieval, redirection, and synthesis, are decoupled, allowing each component to be independently replaced or upgraded, while also avoiding the limitations imposed by a scarcity of long-duration training data.

  • Comprehensive evaluations were conducted using both user studies and physical/kinematic metrics (object jerk, human-object penetration, foot sliding, and human body jerk), with most metrics outperforming the baselines.

Critical weaknesses
  • Despite achieving impressive results in baseline comparisons, neither HOI-Diff nor CHOIS is designed for long-horizon human-object interaction. The lack of comparison against models for long-horizon task interactions, such as those presented in "Human-Object Interaction from Human-Level Instructions" (Wu et al., ICCV 2025), casts doubt on the performance of the proposed method.

  • A notable concern is that the authors did not conduct an ablation study, which makes it unclear how much each individual module contributes to the overall design. In Appendix D, the authors explain that this is due to strong data coupling between the various stages and removing any single stage would cause the entire pipeline to fail. However, the submission lacks comparative analyses within specific stages, such as using random selection or relying solely on structural similarity instead of an LLM-based selection during the retrieval stage. These ablation studies would not disrupt the overall pipeline and could be evaluated using downstream metrics, such as those related to penetration.

  • The results for the 15 sequences reported in Table 1 were selected from a pool of 60, using criteria that overlap with the reported performance metrics. This raises concerns regarding the method's performance.

  • The results shown at the end of the supplemental video have noticeable artifacts, like the pan flipping.

Questions for authors
  • Please compare the proposed method with methods such as ‘Human-Object Interaction from Human-Level Instructions’ (Wu et al., ICCV 2025), which also address long-horizon HOI.

  • Have the authors discussed and explained the advantages of the "retrieval-and-retargeting" paradigm over the "train-to-generate" paradigm?

  • Have the authors conducted an ablation study to analyze the contributions of the individual modules?

Reviewer_TjFaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a training-free, modular system called InterAssembly, which is capable of generating long-duration, multi-objective human-object interaction animations based on a database of short clips (735 clips from HUMOTO). Specifically, the system has three main stages: 1) LLM-driven planning; 2) Interaction-mesh retargeting; and 3) Motion synthesis.

For evaluation, 15 sequences were selected from 60 candidates, averaging 39.1 seconds, 10.7 sub-actions, and up to 7 objects. The authors compared against HOI-Diff and CHOIS using motion quality metrics and a 17-participant user study.

Critical strengths
  1. The motivation is interesting. Long-horizon, multi-object HOI is still largely underexplored. Most existing datasets and generation methods focus on short interactions with a single object. Since collecting long multi-object sequences is expensive, composing existing captured data provides a practical way to scale up the training data.
  2. The system is simple and practical. It requires no task-specific training and costs few LLM tokens per candidate sequence. Its modular design also makes each stage easy to replace or improve. Reusing captured object trajectories likely contributes to the much lower object jerk compared with generative baselines.
Critical weaknesses
  1. The ablations do not tell me whether the proposed pipeline needs these particular components. Showing that a stage cannot be removed is different from showing that the chosen implementation is the right one. For instance, how much does the LLM-based placement improve over rigid alignment or a simple optimizer? Is ARAP retargeting better than IK here? I would also like to see what happens with simpler retrieval and without memory.

  2. I am concerned that contact may be lost during the final synthesis step. MaskControl changes the body motion but leaves the object motion fixed, yet the paper does not measure hand–object contact after this step. The increased foot sliding makes it more important to evaluate the final motion, rather than only the intermediate result.

  3. I am not convinced that eight bounding-box corners adequately describe the object for retargeting, especially for thin or concave shapes. The tetrahedralization is also recomputed every frame. Does its topology ever change, and if so, does that cause visible jitter? The paper should report this as well as the remaining bone-length error after optimization.

  4. The evaluation is confined to HUMOTO. It is also unclear how the 60 target scenes were selected, particularly those derived from seed clips. I would be more confident in the method if it were tested on cluttered scenes, longer sequences, and repeated runs with different LLMs.

Questions for authors
  • I find Table 1 difficult to interpret. InterAssembly’s sequences are much longer and involve more objects, while the baselines use different data, skeletons, and frame rates. Can the authors compare them on the same short interactions? Otherwise the jerk and penetration numbers do not tell us much about relative motion quality.
Reviewer_zFXEOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces InterAssembly, a modular pipeline that generate long-horizon, multi-objects human-object interaction (HOI) animations from short clips in the HOI database. An LLM is responsible for planning action, using structural scene similarity and semantic matching to retrieve and select reference clips, and using collision feedback to adjust object placement. ARAP adapts the clip to the target scene, and then MaskControl converts the joint trajectories into plausible motion. The experiment show that the sequences generated by InterAssembly had an average length of 39.1 seconds and comprised 10.7 sub-actions, which is longer than the single-interaction outputs of the HOI-Diff and CHOIS baselines.

Critical strengths
  1. The method achieved good results in a critical problem. The InterAssembly combines action planning, segment retrieval, scene adaptation, and motion generation into a set of processes. The generated sequence averages about 39 seconds and contains about 10.7 sub-actions, which is beyond the scope of a short-term single interaction.

  2. The design is insightful, where modular design and independent of long-horizon scene-aware HOI data training can benefit from the advancement of each module.

Critical weaknesses
  1. Comparisons with scene-aware interaction methods are Insufficient. The evaluation only has HOI-Diff and CHOIS, which target short HOI while InterAssembly emphasize multi-step interactive assembly under scene conditions. It is recommended that the author compare InterAssembly with related scene-aware HOI methods such as HOSIG,InfBaGel, and LINGO.

  2. Lack of component-level validation. The reported results did not isolate the effects of InterAssembly design choices. The paper also does not provide controlled ablation to quantify the contribution of retrieval, LLM-driven placement, interaction mesh adaption, or MaskControl.

[1] HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception (arXiv:2506.01579, 2025).

[2] InfBaGel: InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement (arXiv:2604.04843, 2026).

[3] LINGO: Autonomous Character-Scene Interaction Synthesis from Human-Level Instructions (arXiv:2410.03187, 2024).

Questions for authors
  1. Can the end-to-end running time of a single sequence be reported? The paper gives the LLM token cost, but does not report the time consumption of each stage. For example, for an output that takes approximately 39 seconds on average, how much average time is required to generate it?

  2. Can the author explain the generation success rate of the pipeline? The paper reports 15 sequences out of 60 candidates. Are the remaining candidates mainly geometric filter failed, or are they not completed due to calculation and rendering budget? I want to know whether these 15 roughly represent the generation success rate, or just the number of samples finally included in the evaluation.

If the author can resolve the weakness and question, I am willing to increase my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.