acceptodds
Under review as a conference paper at ICLR 2027

CompAct: Compositional 3D Human-Object Interaction Synthesis

Abstract

We rarely interact with a single object; we carry, hold, and manipulate several at once. Most methods on 3D interaction synthesis ignore multi-object interactions, as this needs new data capture or augmentation of many (ideally all) possible combinations; however, this does not scale. Our insight is that multi-object interactions are compositions of single-object ones. Thus, we represent interactions at the body-part level through interaction fields (InterFields) that encode contact and proximity. We exploit this in CompAct, a novel Flow-Matching model that, given a text prompt and body and object shape, jointly generates body pose, object pose, as well as body-part and object InterFields. At test time, CompAct predicts proxy single-object interactions and dynamically composes a multi-object one, using classifier-free guidance. To this end, composition exploits blending weights that naturally arise from the per-part InterFields of proxy interactions. Note that this is not a naive linear combination; instead, it is informed by a body prior arising from the classifier-free guidance, so that the composed interaction looks natural. Moreover, note that CompAct uses decoupled timesteps per input part; this supports interaction editing as a downstream application. CompAct is the first model to synthesize multi-object interactions by training only on single-object ones. Experiments on seven single- and multi-object benchmarks show that CompAct outperforms the state of the art in interaction fidelity, and produces multi-object interactions that are perceived as natural looking. Code and models will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human-object interaction, multi-object interaction, interaction representation, contact representation, 3D synthesis, compositional synthesis, flow matching, guided sampling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WG7zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CompAct, a compositional 3D HOI synthesis framework that generates multi-object interactions by combining single-object interactions through body-part InterFields and flow matching without multi-object training data. Experiments demonstrate improvements in interaction synthesis and contact generation over existing methods, supported by ablation studies and perceptual evaluations.

Critical strengths
  1. This paper tackles an important and challenging problem of synthesizing multi-object HOI without requiring multi-object interaction training data. The core idea of composing multiple single-object interactions is meaningful and could be an important step forward for scalable HOI synthesis.
  2. The idea of body-part InterFields and using them to guide the interaction composition are well motivated and specifically designed.
  3. One of the interesting advantages is that the proposed representation naturally supports different conditional generation settings and interaction editing. Figure 2 provides a clear explanation of how these components are integrated.
  4. Experimental results are promising without using multi-object training samples.
Critical weaknesses
  1. Although composing single-object interactions is a promising approach, it is not fully clear how this formulation handles dependencies between multiple interactions that share the same body parts. For example, interactions involving both hands and upper body are often physically coupled. The proposed per-part composition may not fully capture such dependencies, particularly when multiple objects impose conflicting constraints on the same kinematic chain.
  2. The main contribution of this paper is compositional multi-object synthesis. However, in Appendix Table 5, CompAct performs worse than mTRiDi in several reconstruction metrics when both human and object are jointly generated (H, O, I), although it provides better contact gap. This raises a question on whether the method accurately captures the full multi-object interaction distribution.
Questions for authors
  1. In L064-L065, the paper states that a multi-object interaction is a composition of single-object interactions. Could you explain why this assumption is reasonable when multiple interactions can strongly influence each other?
  2. How does CompAct handle cases where two objects require conflicting body configurations? For example, when the same hand or arm needs to interact with two objects simultaneously.
  3. In L272-L275, the paper states that InterFields settle before the human pose during sampling. Why does this happen? Is this an observed behavior of the model or an assumption of the proposed guidance?
  4. In L318-L335, the paper explains that multiple interaction branches can claim the same body part. How does the model ensure that the resulting body configuration satisfies all the interactions when the predictions from these branches are different?
  5. In Appendix Table 5, CompAct performs substantially worse than mTRiDi in MPJPE and object reconstruction error for (H, O, I), while achieving lower contact gap. What causes this difference? Does the compositional guidance prioritize contact satisfaction over the reconstruction of realistic multi-object configurations?
  6. The paper states that the framework supports multi-object interaction synthesis without multi-object training. Does the method also prevent collisions between different objects? How are object-object relationships considered during composition?
  7. The method generates static interaction configurations rather than full interaction sequences. How can the proposed compositional formulation be extended to dynamic HOI where contacts appear and disappear over time?
  8. LLM-estimated contacts can refer papers such as TeHoR (CVPR 2026, Nam et al.), ContactPrompt (arXiv 2026, Jung et al.).
Reviewer_pGrfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CompAct is a flow-matching model for static human–object interactions. It represents body parts and objects with pose tokens and interaction fields, trains on seven single-object datasets, then composes multiple object-conditioned branches at inference using contact masks inferred from text, per-part weights, classifier-free guidance, and an unconditional body prior. It also supports part-level editing.

Critical strengths
  1. The interesting idea is to build multi-object interactions from single-object training data. Assigning contact to body parts gives the composition step a concrete mechanism, rather than simply blending whole-body poses.

  2. The fixed-object results are strong. On HIMO, MPJPE drops from 138.7 to 97.5 mm and contact F1 rises from 56.0% to 69.6% against mTRiDi. The single-object comparisons and ablations also give some support to the representation.

Critical weaknesses
  1. The multi-object result changes substantially when object poses are not given. The main text focuses on fixed-object HIMO, but in Appendix Table 5, joint body–object generation has worse body error, object error, and contact F1 than mTRiDi; contact F1 is only 5.0% versus 16.7%. This is too important to leave mainly in the appendix.

  2. Even with fixed objects, the physical-contact results are mixed. Contact ratio is almost the same as mTRiDi’s (0.46 vs. 0.45), while penetration depth is higher (23.9 vs. 12.3 mm). The improved pose error and contact F1 therefore do not tell the whole story about interaction quality.

Questions for authors
  • Appendix Table 5 looks very different from the fixed-object result in the main text. When the model also has to generate object poses, contact F1 drops to 5.0%. Where does it break down? Please show a few failures from this setting.

  • How sensitive is the composition to mistakes in the text-derived contact masks? In particular, what happens when the wrong hand is assigned, or two objects need the same hand?

  • In the fixed-object HIMO test, contact F1 improves but penetration is worse than mTRiDI’s. Is this caused by a few difficult objects, or does it happen across the test set?

Overall, the results are good, especially when the object poses are given. On HIMO, CompAct substantially improves body error and contact F1 over mTRiDI. The weaker joint-generation result is a real concern, but the problem setting and the fixed-object results make me lean toward acceptance.

Reviewer_B7xaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that static multi-object human-object interactions can be synthesized as compositions of single-object interactions, so that no multi-object training data is needed.

CompAct is a part-wise Flow-Matching MM-DiT that jointly generates 26 body-part poses, per-part signed interaction fields (InterFields), the object pose, and an object InterField, with decoupled per-variable timesteps, kinematic-chain attention, and condition dropout; at test time it runs one branch per object plus an unconditional branch and blends them per body part with weights read from its own predicted InterFields, under self-InterField guidance and LLM-derived contact masks.

Trained on seven single-object datasets, it improves over a retrained TRiDi baseline on GRAB and BEHAVE under three conditionals (15 of 16 error cells), is preferred over multi-object-trained mTRiDi baselines in a 20-participant perceptual study on HIMO, and reports ablations on part granularity, guidance, and InterFields vs. binary contacts.

Critical strengths
  1. A scalable formulation of multi-object HOI as part-level composition of single-object interactions. Instead of collecting multi-object data for every combination, CompAct composes per-object branches with per-part weights that come from its own predicted InterFields, around an unconditional body prior learned through condition dropout (Sec. 3.4; derivation in Appx. B). This is a principled extension of composable diffusion (Liu et al., 2022) from global conditions to body-part-grounded interactions, and it is consequential because it removes the main data bottleneck of multi-object HOI. Decoupled per-part timesteps also give interaction editing for free (Fig. 8).

  2. Clear single-object gains over a strong, matched baseline, and favorable perceptual results. Under TRiDi's exact GRAB+BEHAVE split and best-of-three protocol, CompAct lowers 15 of 16 error cells and raises 7 of 8 contact accuracies against sTRiDi (Table 1), e.g., object vertex error under joint generation 30.96 →\rightarrow 25.56 cm on GRAB and 45.35 →\rightarrow 34.96 cm on BEHAVE. On HIMO, participants preferred CompAct over mTRiDi in 82.2% (whole body) and 85.1% (hands) of comparisons, even though mTRiDi was trained on multi-object data (Sec. 4.3).

Critical weaknesses
  1. The multi-object evidence is limited and mixed, and the headline claims overstate it. Quantitative multi-object comparisons are only against TRiDi-derived baselines. With fixed objects (Table 2), CompAct has better MPJPE/PA/F1 but a larger contact gap (15.3 vs. 3.7 mm), lower contact ratio (0.46 vs. 0.87) than mTRiDi+IK, and nearly twice the penetration depth of mTRiDi (23.9 vs. 12.3 mm). Under joint generation (Table 5, appendix only), CompAct is worse than mTRiDi on MPJPE (238.5 vs. 198.9 mm), object error (347.6 vs. 260.3 mm), and contact F1 (5.0 vs. 16.7%), though it closes contacts better. Multi-object methods discussed in the related work, such as Peng et al. (2026), Dex2HOI (Pratikaki et al., 2026), and the zero-shot HOI-PAGE (Li & Dai, 2026), are not compared, and no variance is reported. Meanwhile, the abstract states that "experiments on seven single- and multi-object benchmarks show that CompAct outperforms the state of the art in interaction fidelity," although the seven datasets are training sets and quantitative comparisons are on GRAB, BEHAVE, and HIMO. This matters because multi-object synthesis is the paper's main claim.

    • Suggestion: compare with the multi-object methods discussed in the related work where feasible, report variance, and align the abstract with the benchmarks actually used for quantitative comparison.
  2. The core composition mechanism is not isolated by the ablations, and some ablation numbers are hard to reconcile. The ablations (Table 3) vary part granularity, guidance, and InterFields vs. binary contacts, but not the composition rule itself: there is no comparison with fixed-weight composable diffusion (Liu et al., 2022), naive per-part stitching, composition without the unconditional branch, or the object-freeze heuristic. The granularity results are also non-monotonic: 8 parts gives lower single-object MPJPE (118.6 vs. 119.9 mm) and contact gap (9.4 vs. 10.1 mm) and lower HIMO MPJPE (102.1 vs. 109.3 mm) than the full 26-part model, and 16 parts is worse than 8 parts on most columns. The full model's HIMO numbers in Table 3 (MPJPE 109.3, Gap 18.7) differ from Table 2 (97.5, 15.3) for the same fixed-object setting. Finally, the LLM used to produce the contact masks is not named, even though test-time masks depend on it (exact full-mask agreement only ~27%, Appx. C). Without these, it is unclear whether the InterField-weighted composition, rather than the single-object model quality, drives the multi-object results.

    • Suggestion: ablate the composition rule itself (fixed-weight composable diffusion, naive per-part stitching, no unconditional branch, no object freeze), explain the non-monotonic granularity results and the Table 2/3 discrepancy, and name the LLM used for the contact masks.
Questions for authors

Clarifications (could change my assessment):

  1. How does InterField-weighted composition compare with fixed scalar weights (Liu et al., 2022) and with naive per-part stitching of the single-object branches on HIMO, using the same trained model?
  2. Why do the full-model HIMO numbers differ between Table 3 (MPJPE 109.3 mm, Gap 18.7 mm) and Table 2 (97.5 mm, 15.3 mm)? Are they from different checkpoints?
  3. Under joint generation on HIMO (Table 5), CompAct is worse than mTRiDi on MPJPE, object error, and contact F1. How should this be reconciled with the claim of state-of-the-art interaction fidelity? Why is penetration depth on HIMO roughly twice that of mTRiDi (23.9 vs. 12.3 mm), and does self-InterField guidance contribute to it?
  4. Which LLM produces the contact masks, and how do the multi-object results change with the ground-truth (oracle) masks mentioned in Sec. 4?

I am open to revisiting my rating if the rebuttal addresses these points well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.