acceptodds
Under review as a conference paper at ICLR 2027

Harnessing Coupled Stream Completion for Human–Object Interaction Modeling

Abstract

Text-conditioned human-object interaction (HOI) generation requires body motion, object trajectories & rotations, and hand articulation to remain coordinated. These components differ in scale and dynamics, but must agree on contact, relative pose, and timing. A shared representation may limit the distinct structure of each stream, while independent generation prevents each stream from responding to changes in the others. Latent supervision alone also does not directly constrain contact after decoding. We propose TRACE, a continuous latent framework that keeps stream states separate and couples their updates. TRACE encodes body, object, and hand motion into separate latents and predicts each stream velocity from the complete current interaction state. Geometric losses on decoded motion further constrain contact and object-relative motion over time. The same model supports completion of any single absent stream from the other two. Frozen flow features also serve as input to a language model for HOI understanding. Experiments on InterAct, OMOMO, and BEHAVE show that joint completion training improves generation and that frozen flow features improve understanding over raw-motion encoding. On InterAct, TRACE achieves the highest contact precision, recall, and F1 among the compared methods.

Primary area:
generative models
Keywords:
Human-Object Interaction, Motion Generation, Motion Understanding, Generative Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tgFdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper introduces TRACE, a framework for jointly generating body, hand, and object motion conditioned on text and object geometry.
  • The authors jointly train full interaction generation with three single-stream completion tasks and connect frozen features from the generative model to a language model for interaction description.
  • The paper provides extensive experimental validation on InterAct, OMOMO, and BEHAVE, covering generation, completion, and interaction description, with ablation studies examining stream modeling, latent representations, and geometric supervision.
Critical strengths
  • The model architecture is well designed and closely matches the challenges of coordinating body, hand, and object trajectory.
  • The experimental results show clear improvements in contact consistency, and extensive ablation studies support the effectiveness of the key design choices.
  • The visualizations are clear and informative, effectively illustrating the quality of the generated interactions.
Critical weaknesses
  • The novelty of the coupled-flow formulation could be further clarified.Prior work such as LIGHT already jointly models body, hand, and object streams and enables cross-stream information exchange.
  • The implementation details of the stream-modeling ablations in Table 3 are insufficiently described.
Questions for authors
  • Could the authors provide more architectural details for the ablation variants in Table 3, and clarify how they differ from the proposed coupled model?
  • Could the authors discuss the balance between contact consistency and text–motion alignment, given that the R-Precision scores in the main results table are somewhat lower than those of some baselines?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors for Reviewer tgFd

We thank the reviewer for thoughtful and insightful comments. We address the concerns below:

1. Methodological Distinction (W1): LIGHT exchanges information across body, hand, and object streams and uses different denoising rates to guide generation in motion space. TRACE instead learns a shared flow over separate continuous latents. Each stream has its own velocity, but all three velocities depend on the full current state. We train the same flow for full generation and masked completion. For example, with observed hand and object motion, it holds both streams fixed and updates only the body. Contact and object-relative losses supervise the decoded motion. Table 4 in the paper shows that continuous latents reduce FID from 0.228 to 0.199 compared with raw motion, and Table E evaluates completion under fixed observations.

2. Ablation Configurations (W2, Q1): All three variants use the same frozen split VAE, text and object conditions, and training objectives. Unified concatenates the three latents along the feature dimension and processes them with one Transformer. Independent keeps separate branches, each reading only its own stream state. Coupled keeps stream-specific projections and exchanges information through joint attention. From Unified to Coupled, Contact F1 rises from 0.760 to 0.799, while R@1 increases from 0.400 to 0.404. We will add these architecture details to the revised manuscript.

3. Contact and Text Alignment (Q2): In Table 1 of the paper, TRACE achieves higher Contact F1 than LIGHT (0.799 vs. 0.627) and lower R@1 (0.404 vs. 0.421). Within TRACE, adding geometric supervision in Table 5 improves both Contact F1 (0.715 to 0.799) and R@3 (0.736 to 0.744). We will discuss the retrieval gap alongside the contact gains and distinguish the comparison across methods from the effect of geometric supervision within TRACE.

We thank the reviewer for the constructive feedback. If the reviewer has any further questions or suggestions, we are happy to discuss them.


Dear reviewer, the authors have responded to your comment. Could you please take a look? Let us know if you have further concerns.

Thanks for the author's response. The rebuttal has addressed most of my concerns.

Therefore, I will maintain my original rating (weak accept).

Reviewer_UaCiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

TRACE encodes body, object, and hand motion in separate continuous latents, then couples their flow updates through the full evolving interaction state. Decoded-space geometric losses encourage contact and object-relative consistency. One model also completes a missing stream from the other two; frozen flow features are used for HOI-language understanding. The main evaluation is on InterAct, with additional OMOMO and BEHAVE experiments。

Critical strengths
  1. Modeling body, hands, and object motion as separate streams makes sense, given their different dynamics. The independent and unified ablations help show why the streams also need to interact.

  2. The contact results are strong. On InterAct, TRACE^joint reaches 0.806 contact F1, compared with 0.667 for ROG, the strongest listed baseline on that metric.

Critical weaknesses
  1. The paper takes too long to explain its main idea. Separate encoders and decoders handle body, hands, and object motion, while a transformer lets the three latent streams interact. This is repeated throughout the introduction and method. Equations 5–6 recast the interaction in Jacobian notation, but I could not see how that formulation changes the model beyond joint attention. Shortening the related work and metrics sections would also make the contribution easier to identify.

  2. Text alignment is weak, and the paper doesn’t discuss it. In Table 1, TRACE’s R@1 is 0.404, second-lowest of the seven methods. InterDiff reaches 0.501, CHOIS 0.439, Text2HOI 0.428, and even HOI-Diff 0.413; only ROG is lower. R@2 and R@3 follow the same pattern. Penetration (0.117) is also worse than HOI-Diff, Text2HOI, and ROG. The text focuses on contact metrics, MM Dist, and FSR and never mentions this. For a text-conditioned generation method, the authors should explain why alignment drops.

  3. The evaluation leaves some basic questions open. There is no unseen-object test, user study, or supplementary video. The contact numbers are strong, but it is hard to judge from them alone whether the generated interactions look convincing or transfer to new objects.

Questions for authors
  • InterDiff is originally a motion-history-conditioned prediction model. The paper states that its history encoder is replaced with a text encoder, but gives too little detail to assess this adaptation. What inputs remain at inference time, how is its physics-based correction module used without motion history, and how is the text encoder trained? Since this adapted baseline has the highest R@1 in Table 1, these details and its implementation should be provided.
  • How does completion compare quantitatively with OMOMO or DiffGrasp on body completion?
7 comments
AuthorsOpenReview ↗

Rebuttal by Authors for Reviewer UaCi

We thank the reviewer for thoughtful and insightful comments. We address the concerns below:

1. Coupled Flow (W1): Equations (5)–(6) distinguish a velocity that depends on its own stream from one that depends on the full interaction state. TRACE implements the latter through joint attention with stream-specific projections. The separate latent states let us train the same flow for full generation and masked completion. For example, when the hand and object trajectories are given, their latents stay fixed throughout sampling and condition each update of the missing body stream. The decoded losses supervise the resulting contact and object-relative geometry. We will use this example to explain the formulation and shorten the introduction, related work, and metrics discussion.

2. Text Alignment and Penetration (W2):

(1). For Text Alignment: Such competitive result of these baselines, a similar observation was also reported in previous work [1,2]. Replacing the motion-history encoder with a text encoder makes the adapted architecture similar to MDM-based generators. We hypothesize that local temporal attention promotes smooth motion by emphasizing dependencies between nearby frames, which may contribute to its competitive performance. Besides, TRACE has lower R@1 than adapted InterDiff (0.404 vs. 0.501) and lower MM Dist (2.256 vs. 2.461). R-Precision measures relative retrieval rank, while MM Dist averages the distance between matched pairs. So, the lower text alignment cannot related to low-quality generation.

(2). For Penetration: It should be read alongside contact accuracy, since missing intend contacts can also reduce collisions. A hand can avoid penetrating a handle by missing the grasp. HOI-Diff has lower penetration than TRACE (0.103 vs. 0.117) and lower contact recall (0.447 vs. 0.810). We will discuss both the retrieval gap and the contact–penetration balance explicitly.

3. Qualitative Evidence and Evaluation Scope (W3):

(1). Demos. We provide four supplementary video clips with their corresponding input prompts. Please download them from the supplementary materials.

(2). User Study. We have 7 participants compared 10 pairs of TRACE and LIGHT sequences. TRACE was preferred in 57 of 70 judgments. We will build a complete user study in the revised manuscript.

(3). Unseen Object Study. We additionally evaluated generalization on the original split of OMOMO [3], which have unseen object of the test set. In this setting, TRACE achieves 0.312 R@1, 0.101 FID, and 0.873 contact recall, versus 0.264, 0.129, and 0.800 for LIGHT with the same text and object geometry inputs.

Table 1: Results on the original split of OMOMO.

MethodGiven inputR@1 ↑R@2 ↑R@3 ↑FID ↓MM Dist. ↓Diversity →FSR ↓Pene. ↓Contact →Contact precision ↑Contact recall ↑
LIGHTText + object geometry0.264 ± 0.0050.470 ± 0.0040.632 ± 0.0040.129 ± 0.0131.766 ± 0.0287.375 ± 0.0520.023 ± 0.0060.115 ± 0.0100.238 ± 0.0140.840 ± 0.0140.800 ± 0.016
OMOMOGT object motion + geometry0.288‾\underline{0.288} ± 0.0040.507‾\underline{0.507} ± 0.0030.664‾\underline{0.664} ± 0.0030.115‾\underline{0.115} ± 0.0101.691‾\underline{1.691} ± 0.0217.395 ± 0.0470.018‾\underline{0.018} ± 0.0050.106‾\underline{0.106} ± 0.0080.256 ± 0.0120.880‾\underline{0.880} ± 0.0120.835‾\underline{0.835} ± 0.014
TRACEText + object geometry0.312 ± 0.0030.544 ± 0.0020.696 ± 0.0020.101 ± 0.0071.616 ± 0.0147.415 ± 0.0420.013 ± 0.0040.097 ± 0.0060.274 ± 0.0100.890 ± 0.0100.873 ± 0.012
AuthorsOpenReview ↗

Rebuttal by Authors for Reviewer UaCi

4. InterDiff Implementation (Q1): At inference, the adapted InterDiff generates the full HOI sequence from Gaussian noise conditioned on the text prompt and object geometry. Sampling bypasses the original physics correction module, which relies on observed motion history. The text encoder uses pretrained CLIP weights and remains frozen during training. Training updates the text projection and denoiser, with the input and output projections resized to match the representation of the dataset.

5. Body Completion Baseline (Q2): We evaluated OMOMO (Stage 2) and TRACE on the dataset OMOMO using a common evaluator. OMOMO (Stage 2) receives two ground-truth hand anchors, and TRACE receives the observed hand and object streams. Table 2 below reports error over the body joints error (Joint error) and root translation error (Root error).

Tabel 2: Body completion errors for OMOMO and TRACE.

MethodObserved inputJoint error ↓Root error ↓
OMOMOTwo GT hand anchors57.1292.69
TRACEHand and object motion50.8983.30

We thank the reviewer for the constructive feedback. If the reviewer has any further questions or suggestions, we are happy to discuss them.


[1] L.-A. Zeng, G. Huang, Y.-L. Wei, S. Gu, Y.-M. Tang, J. Meng, and W.-S. Zheng, “ChainHOI: Joint-based kinematic chain modeling for human-object interaction generation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025.

[2] Z. Wang et al., “Unleashing guidance without classifiers for human-object interaction animation,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2026.

[3] J. Li, J. Wu, and C. K. Liu, “Object motion guided human motion synthesis,” ACM Trans. Graph., vol. 42, no. 6, 2023.

Dear reviewer, the authors have responded to your comment. Could you please take a look? Let us know if you have further concerns.

Reviewer_K54TOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents TRACE, a framework for text-conditioned human–object interaction generation. It separately encodes body, hand, and object motion into continuous latent spaces and coordinates their evolution through a coupled flow transformer. Geometric losses supervise contact and object-relative motion after decoding. The model jointly supports generation and single-stream completion, while its frozen features are connected to a language model for interaction captioning. Experiments on InterAct, OMOMO, and BEHAVE evaluate generation, completion, and understanding.

Critical strengths
  1. The framework supports generation and three stream-completion tasks within a shared generative backbone. Its frozen features also improve captioning over raw-motion encoding under matched language-model settings, demonstrating useful reuse of the learned representation.
  2. TRACE achieves the strongest contact precision, recall, and F1 on InterAct among the compared methods, alongside a low foot-skating ratio. The geometric-loss ablations provide supporting evidence that the proposed constraints improve interaction consistency.
Critical weaknesses
  1. The methodological novelty beyond established body/hand decomposition, such as TalkSHOW and HumanTOMATO, remains unclear. Table 3 shows only a small R@1 gain over unified modeling (0.400 to 0.404); although contact F1 improves, insufficient details on architectural differences and parameter matching make the contribution of coupling difficult to isolate.
  2. The performance advantage is inconsistent across tasks and metrics. Despite strong contact scores, Table 1 reports weaker R-Precision and FID than leading baselines, while Table 2 shows mixed understanding results, with BLEU-1, ROUGE, and BERTScore below HOIGPT.
Questions for authors
  1. The methodological distinction between the proposed coupled flow and existing body/hand decomposition or standard joint attention over separate streams needs clearer explanation.
  2. The architectural differences between the Unified and Coupled variants in Table 3 should be specified, including whether they are matched in parameter count, latent encoders, routing heads, and training objectives.
  3. The stronger contact scores but weaker R-Precision and FID in Table 1, together with the mixed results relative to HOIGPT in Table 2, require clearer interpretation.
  4. The evaluation protocol should clarify whether the splits test unseen objects or interaction types. The generalization supported by separate training on InterAct and its constituent datasets should also be stated more precisely.
  5. The effectiveness of the geometric losses in addressing interpenetration needs further explanation, given the apparent residual artifacts in the qualitative examples and the worse penetration scores than several baselines in Table 1.
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors for Reviewer K54T

We thank the reviewer for thoughtful and insightful comments. We address the concerns below:

1. Methodological Distinction (W1, Q1): Compared to the given methods, TalkSHOW [1] and HumanTOMATO [2], TRACE differs in three respects. (1) Both methods only generate human motion, while TRACE also generates the rigid object trajectory from text and canonical object geometry. Dynamic HOI additionally couples articulated body and hand motion with moving rigid objects: small spatial or temporal errors can produce missed grasps, sliding contact, or penetration, even when each component appears individually plausible. (2) For modeling, TRACE represents object rotation and translation in a separate latent stream. At each flow step, the body and hand velocities depend on the current object state, and the object velocity depends on both human streams. (3) For supervision, beyond human pose, TRACE supervises contact on decoded motion. It constrains where human anchors lie in the object frame and how those positions change during a grasp.

2. Ablation Configurations (W1, Q2): The variants use the same frozen split VAE, text/object conditions, flow-matching objective, and decoded geometric objectives. The "Unified" have feature-wise concatenation of the three latents, processed through one Transformer pathway. The "Independent" separate stream pathways without cross-stream state exchange. The "Coupled" set stream-specific projections with joint attention across streams. Besides, we scaled up the "Unified" Transformer to make its trainable parameter comparable to other two settings. We will add these explanation on the revised manuscript. Besides, the R@1 gains is modest (0.400 to 0.404), while contact recall improves from 0.764 to 0.810 and F1 from 0.760 to 0.799.

3. Quantitative Results (W2, Q3):

(1) Generation Results: For the such competitive result of Interdiff, a similar observation was also reported in previous work [3,4], where Interdiff was adapted as text-conditioned and achieved strong performance. Local temporal attention may help InterDiff produce smoother motion across nearby frames. TRACE has lower R@1 than the baselines, but have a better FID 0.188 and higher contact F1 0.806. We will add these discussion in the revised manuscript.

(2) Captioning Results: Captioning evaluates whether generative features provide reusable interaction information. Under the same language-model setting, frozen flow features improve every reported metric over raw-motion encoding (Table 2 in paper). Against HOIGPT, TRACE leads on R@1, MM Dist, BLEU@4, and CIDEr on InterAct, while trailing on other metric. Results also vary across OMOMO and BEHAVE. We will scope our conclusion to representation utility and acknowledge the mixed external comparisons.

4. Evaluation Scope (Q4): In the submitted paper, we follow the official dataset splits of InterAct, without an additional object or interaction protocol. OMOMO and BEHAVE models are trained separately on their respective training sets. We will clarify that these results establish performance across benchmark settings.

5. Geometric Loss & Penetration (Q5): Low penetration should be interpreted alongside contact fidelity, since missing intend contacts can also reduce collisions. A hand can hover beside a handle without penetrating it and still miss the grasp entirely. In table 1 of the paper, HOI-Diff and Text2HOI have lower penetration (0.103/0.105), but their contact ratios (0.084/0.102) are well below ground truth. Our contact loss encourages contact between hands and objects, effectively improving the quality of the generated contacts.

We thank the reviewer for the constructive feedback. If the reviewer has any further questions or suggestions, we are happy to discuss them.


[1] H. Yi et al., “Generating holistic 3D human motion from speech,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023.

[2] S. Lu et al., “HumanTOMATO: Text-aligned whole-body motion generation,” arXiv preprint arXiv:2310.12978, 2023.

[3] L.-A. Zeng, G. Huang, Y.-L. Wei, S. Gu, Y.-M. Tang, J. Meng, and W.-S. Zheng, “ChainHOI: Joint-based kinematic chain modeling for human-object interaction generation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025.

[4] Z. Wang et al., “Unleashing guidance without classifiers for human-object interaction animation,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2026.

Dear reviewer, the authors have responded to your comment. Could you please take a look? Let us know if you have further concerns.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal to Reviewer K54T

Dear Reviewer K54T,

We hope this message finds you well. We would like to gently remind you that the discussion period closes today.

In our rebuttal, we explained how TRACE couples body, hand, and object motion and how we made the Unified and Coupled models comparable in size for Table 3. We also clarified the benchmark splits and the separate training of the OMOMO and BEHAVE models.

We would appreciate a brief reply. If our response has addressed your concerns to some extent, we hope you will consider raising our score accordingly.

The Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.