acceptodds
Under review as a conference paper at ICLR 2027

ULIFlow: Unified Latent Human-Object Interaction Flow

Abstract

Text-conditioned 3D human–object interaction (HOI) generation remains challenging because individually plausible human and object motions do not necessarily form a coherent interaction. We argue that a key limitation lies in conventional skeleton-based representations, which compactly describe body kinematics while leaving surface-level interaction geometry largely implicit, often requiring additional contact-specific regularizers. We propose ULIFlow, a framework that explicitly models continuous human–object relations within a unified latent space. To this end, we derive continuous marker–object relations from sparse body-surface markers and treat them as an explicit interaction modality alongside human motion and object dynamics. A Mixture-of-Transformers (MoT) autoencoder is then designed to jointly model these heterogeneous streams while preserving their modality-specific structures. Importantly, we let the relation stream directly participate in human and object reconstruction, encouraging the shared latent representation to preserve both individual dynamics and the interaction geometry that coordinates them. To further achieve variable-length generation, we introduce a blockwise Flow Forcing scheme with global sequence-progress conditioning, enabling progressive continuation from fixed-length training windows while maintaining consistent long-horizon dynamics. Extensive experiments demonstrate that ULIFlow consistently outperforms prior work in overall generation quality and human–object coordination, while preserving coherent interactions over requested durations beyond the training horizon.

Primary area:
generative models
Keywords:
3D human-object interaction generation, text-conditioned motion generation, human motion representation, flow matching, diffusion forcing, variable-length motion generation, multimodal generative modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_y9xGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ULIFlow, a framework for text-conditioned 3D human–object interaction (HOI) generation with variable length. The authors derive continuous marker–object relations (e.g., relative offsets, distances, and their temporal differences) from body-surface markers and treat them as an additional modality alongside human and object motion. A Mixture-of-Transformers (MoT) autoencoder jointly encodes the three modalities into a unified latent space, and a conditional latent DiT is trained with a blockwise Flow Forcing scheme and sequence-progress conditioning to generate HOI sequences beyond the fixed training window. Experiments on the InterAct dataset show improvements over prior methods in generation quality and human–object penetration.

Critical strengths
  • This paper introduces a new marker-based representation that explicitly models interaction geometry, which has not been explored in previous methods. Specifically, the authors explicitly model continuous human–object relations within a unified latent space and introduce a Mixture-of-Transformers autoencoder that jointly encodes human, object, and relation modalities.
Critical weaknesses
  1. The comparison in Table 1 is unfair, and some results are questionable.
  • Several compared methods are not trained on InterAct (e.g., CHOIS and HOI-Diff are trained on OMOMO and BEHAVE, respectively), which leads to an out-of-distribution (OOD) evaluation, whereas the proposed method is trained and tested on InterAct.
  • The reproduced results seem unusual. For example, as shown in Table 1, HOI-Diff achieves very poor performance (0.134 Top-1 R-Precision and 14.445 FID), which is much worse than CHOIS. In contrast, the results reported in the LIGHT paper show only a small performance gap between HOI-Diff and CHOIS.
  1. The authors claim that their method supports variable-length generation, but they do not provide sufficient experiments to support this claim.
  • All baselines in Table 2 and Fig. 3 are single-window methods, and it is unclear how their long sequences and "continuation boundaries" are obtained. A continuation baseline (e.g., InterAct with sliding-window inpainting) is needed for a fair comparison.
  • In the ablations, the variant without blockwise forcing achieves better FID than the full model in several settings (Table C, 128 frames: 6.837 vs. 7.712; Table D: 4.853 vs. 4.917 and 3.554 vs. 3.800), so the benefit of the proposed schedule is mainly reflected in jerk statistics.
  1. All models are trained only on the InterAct dataset. Although Table B reports results on the BEHAVE and OMOMO subsets, these are subsets of InterAct evaluated with the model trained on the full InterAct dataset. It would be better if the authors also trained and evaluated their method on the original OMOMO and BEHAVE benchmarks following their standard protocols, so that the results can be directly compared with the numbers reported in prior work.
Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Critical Weaknesses (Part 1)

We sincerely thank Reviewer y9xG for recognizing the value of explicitly modeling continuous human–object relations within a unified latent space.

We address the concerns and questions below.

W1. The comparison...

  • Regarding differences in training data sources: To address this concern, we retrained our model on OMOMO and BEHAVE and conducted comparisons on the two datasets. Specifically, we evaluate HOI-Diff on BEHAVE and CHOIS on OMOMO, alongside InterAct, LIGHT, and ULIFlow. Within each comparison, all methods use the same configuration.

Comparison on BEHAVE

MethodFID ↓R@1 ↑R@2 ↑R@3 ↑MM-Dist ↓
HOI-Diff28.686 ± 0.1820.288 ± 0.0230.446 ± 0.0100.565 ± 0.0268.650 ± 0.037
InterAct3.985 ± 0.2210.627 ± 0.0250.842 ± 0.0340.912 ± 0.0175.969 ± 0.062
LIGHT4.318 ± 0.1660.635 ± 0.0400.808 ± 0.0310.896 ± 0.0196.103 ± 0.051
Ours (Full)3.236 ± 0.1020.671 ± 0.0380.863 ± 0.0060.925 ± 0.0115.551 ± 0.050
Ours (Retrained)3.307 ± 0.0450.717 ± 0.0190.854 ± 0.0070.931 ± 0.0115.506 ± 0.032

Comparison on OMOMO

MethodFID ↓R@1 ↑R@2 ↑R@3 ↑MM-Dist ↓
CHOIS10.176 ± 0.0190.647 ± 0.0120.880 ± 0.0130.957 ± 0.0055.352 ± 0.025
InterAct3.176 ± 0.1930.774 ± 0.0010.951 ± 0.0180.991 ± 0.0044.013 ± 0.036
LIGHT1.599 ± 0.0980.781 ± 0.0080.958 ± 0.0120.993 ± 0.0053.860 ± 0.055
Ours (Full)1.081 ± 0.0900.801 ± 0.0140.965 ± 0.0090.994 ± 0.0023.584 ± 0.013
Ours (Retrained)1.212 ± 0.1000.797 ± 0.0170.958 ± 0.0100.993 ± 0.0063.647 ± 0.021

As shown in the tables, training on the full InterAct dataset provides clear benefits compared with training only on OMOMO and BEHAVE. Nevertheless, even when trained only on OMOMO and BEHAVE, our method still achieves better overall performance than other methods trained on the full InterAct dataset.

  • Regarding the unusual reproduced results: Generation-quality evaluation relies on a shared evaluator, which requires a consistent human-motion input representation. However, LIGHT uses a 52-joint skeleton, HOI-Diff uses 22 joint positions, and CHOIS uses joint positions and rotations decoded through SMPL-X, whereas InterAct and ULIFlow use body-surface markers. We therefore adapt baseline outputs to the common 77-marker representation before evaluation, a process that can introduce conversion errors. For the baseline comparisons, LIGHT adapts specific components of the compared methods and retrains a joint-based evaluator, while we reuse InterAct’s publicly released marker-based evaluator. Consequently, differences in representation adaptation and evaluator training lead to differences in the reported results.
AuthorsOpenReview ↗

Response to Critical Weaknesses (Part 2)

W2. The authors claim that...

  • Regarding the long-sequence performance comparison: To address this concern, we extend InterAct with sliding-window inpainting to construct a continuation baseline. Specifically, each new 128-frame window is conditioned on the final 32 generated frames of the preceding window. For a matched comparison, we evaluate both methods on long sequences from the dataset, with the generation length set to 384 frames. Across the rollout, we evaluate non-overlapping 128-frame segments from the early, middle, and late stages, and we further examine 64-frame neighborhoods centered on InterAct’s actual continuation boundaries at positions 128, 224, and 320.

Early stage (frames 1–128)

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
InterAct9.786 ± 0.6120.656 ± 0.0286.239 ± 0.03069.38 ± 1.3447.55 ± 1.48
Ours5.645 ± 0.1750.667 ± 0.0386.424 ± 0.03874.95 ± 1.1957.32 ± 1.78

Middle stage (frames 129–256)

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
InterAct10.046 ± 1.1540.587 ± 0.0476.738 ± 0.14470.15 ± 2.2646.03 ± 0.75
Ours5.905 ± 0.1520.667 ± 0.0286.323 ± 0.00885.87 ± 0.3966.23 ± 1.17

Late stage (frames 257–384)

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
InterAct9.862 ± 0.9530.569 ± 0.0337.108 ± 0.03570.27 ± 2.3945.03 ± 1.61
Ours7.245 ± 0.0300.660 ± 0.0246.539 ± 0.03180.95 ± 0.2662.75 ± 1.22

Boundary at position 128

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
InterAct9.566 ± 1.9790.538 ± 0.0306.971 ± 0.09767.62 ± 1.2244.57 ± 1.05
Ours4.523 ± 0.1210.653 ± 0.0376.483 ± 0.03280.94 ± 1.0561.31 ± 0.84

Boundary at position 224

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
InterAct8.368 ± 1.0070.566 ± 0.0126.920 ± 0.14067.64 ± 2.2942.08 ± 1.39
Ours5.323 ± 0.1660.608 ± 0.0376.680 ± 0.03784.76 ± 0.6965.33 ± 0.98

Boundary at position 320

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
InterAct10.510 ± 1.4660.528 ± 0.0437.251 ± 0.05665.66 ± 3.8240.72 ± 3.40
Ours6.003 ± 0.1170.604 ± 0.0486.848 ± 0.09780.91 ± 0.4363.18 ± 0.23

Here, FID, R@1, and MM-Dist assess generation quality, while Body F1 and Hand F1 assess full-body and hand–object contact quality, respectively.

Across all evaluated settings, our method exhibits a clear overall advantage over InterAct with sliding-window inpainting. These results demonstrate the effectiveness of our design for long-horizon generation.

AuthorsOpenReview ↗

Response to Critical Weaknesses (Part 3)

  • Regarding the effectiveness of temporal modeling designs: To clarify how our temporal modeling designs affect long-horizon generation, we compare the full model with variants without blockwise forcing and without sequence-progress conditioning. Specifically, we generate 384-frame rollouts using long sequences from the dataset and separately evaluate their early, middle, and late 128-frame segments. We also evaluate 64-frame neighborhoods centered at positions 128, 224, and 320, matching the continuation-boundary evaluation above.

Early stage (frames 1–128)

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
Ours w/o progress7.358 ± 0.2250.674 ± 0.0166.168 ± 0.03470.73 ± 0.2153.47 ± 0.85
Ours w/o blockwise5.680 ± 0.0660.674 ± 0.0166.394 ± 0.00877.06 ± 0.9559.18 ± 1.58
Ours (Full)5.645 ± 0.1750.667 ± 0.0386.424 ± 0.03874.95 ± 1.1957.32 ± 1.78

Middle stage (frames 129–256)

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
Ours w/o progress7.198 ± 0.3070.587 ± 0.0536.728 ± 0.07369.02 ± 0.8951.49 ± 2.47
Ours w/o blockwise6.018 ± 0.0190.646 ± 0.0366.373 ± 0.02884.96 ± 0.5465.86 ± 0.53
Ours (Full)5.905 ± 0.1520.667 ± 0.0286.323 ± 0.00885.87 ± 0.3966.23 ± 1.17

Late stage (frames 257–384)

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
Ours w/o progress7.537 ± 0.0640.632 ± 0.0266.684 ± 0.05974.20 ± 1.2455.55 ± 1.32
Ours w/o blockwise7.665 ± 0.0540.628 ± 0.0066.583 ± 0.04679.65 ± 1.2561.42 ± 1.42
Ours (Full)7.245 ± 0.0300.660 ± 0.0246.539 ± 0.03180.95 ± 0.2662.75 ± 1.22

Boundary at position 128

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
Ours w/o progress5.938 ± 0.1360.562 ± 0.0316.749 ± 0.01168.30 ± 0.5350.66 ± 0.41
Ours w/o blockwise4.758 ± 0.1410.705 ± 0.0336.368 ± 0.01082.99 ± 0.2463.43 ± 2.65
Ours (Full)4.523 ± 0.1210.653 ± 0.0376.483 ± 0.03280.94 ± 1.0561.31 ± 0.84

Boundary at position 224

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
Ours w/o progress6.054 ± 0.2760.559 ± 0.0436.827 ± 0.05768.68 ± 1.7651.17 ± 3.17
Ours w/o blockwise5.479 ± 0.1480.608 ± 0.0266.692 ± 0.03883.83 ± 0.5965.04 ± 1.12
Ours (Full)5.323 ± 0.1660.608 ± 0.0376.680 ± 0.03784.76 ± 0.6965.33 ± 0.98

Boundary at position 320

MethodFID ↓R@1 ↑MM-Dist ↓Body F1 (%) ↑Hand F1 (%) ↑
Ours w/o progress6.520 ± 0.3110.583 ± 0.0736.868 ± 0.05472.79 ± 1.6754.59 ± 0.38
Ours w/o blockwise6.245 ± 0.0620.576 ± 0.0126.862 ± 0.07778.90 ± 1.2661.35 ± 1.95
Ours (Full)6.003 ± 0.1170.604 ± 0.0486.848 ± 0.09780.91 ± 0.4363.18 ± 0.23

As shown in the tables, within frames 1–128, which match the training-window length, our temporal modeling designs introduce minor performance trade-offs. As generation extends beyond this window, however, these designs consistently improve both generation quality and human–object coordination. These results demonstrate the effectiveness of our temporal modeling designs for generation beyond the training-window length.

Reviewer_PTqEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Existing text-conditioned 3D human-object interaction (HOI) generation methods often suffer from inaccurate contact modeling due to the limitations of skeleton-based representations, as well as spatial misalignment caused by insufficient coordination between human and object motions. To address these issues, the authors explicitly model human, object, and relation representations and employ a Mixture-of-Transformers (MoT) architecture to preserve modality-specific characteristics while learning coordinated motion. Furthermore, the proposed Blockwise Flow Forcing and sequence-progress conditioning enable variable-length HOI generation beyond the fixed training window.

Critical strengths

S1. The problem definition is clear and well motivated. The paper addresses important challenges in existing HOI generation methods, including insufficient human-object coordination, reliance on contact-specific heuristic constraints, and difficulties in long-horizon generation with fixed-length training windows. The proposed use of explicit relation modeling and Blockwise Flow Forcing provides an intuitive approach to addressing these challenges.

S2. Extensive experiments demonstrate meaningful improvements over prior baselines. The proposed method achieves notable improvements over existing baselines in several metrics, particularly FID, penetration ratio, and long-horizon generation quality. The ablation studies also provide evidence supporting the overall effectiveness of the MoT architecture, relation pathway, and Blockwise Flow Forcing.

Critical weaknesses

Major weaknesses

W1. The necessity of the explicit relation stream and the sources of its performance improvements are not sufficiently disentangled. Although Tables 1, 5, and G demonstrate the effectiveness of the overall relation pathway, the necessity of an explicit relation stream remains unclear, as RR is deterministically derived from HH and OO. The w/o relation branch ablation simultaneously removes relation tokenization, reconstruction loss LRL_R, and relation-coupled decoding. Consequently, it is difficult to determine whether the improvements stem from the explicit relation representation, additional supervision, or decoder coupling.

W2. The methodological novelty and positioning relative to prior work are not sufficiently clear. Several key components, including MoT, body-surface markers, and human-object geometric relations, have been explored in prior work. Thus, the novelty appears to lie primarily in their specific integration rather than the individual components. However, the contribution statements do not clearly distinguish these aspects, and the limited component-wise ablations further obscure the sources of the improvements.

W3. Direct baseline comparisons for contact quality are insufficient. Although Table 4 compares penetration against InterAct and LIGHT, lower penetration alone does not necessarily indicate better interaction quality, as it may result from excessive human-object separation. The Appendix reports contact recall, hand recall, contact distance, and unsupported object travel, but only for internal ablations. Comparing these metrics against InterAct and LIGHT would provide stronger evidence for the claimed improvements in human-object coordination.

Minor weaknesses

W4. The related work positioning regarding coupled HOI modeling is incomplete. The Introduction highlights the challenges of coupled HOI generation due to differing human-object kinematics. However, DecHOI (CVPR 2026) directly addresses these challenges through decoupling. Discussing DecHOI and clarifying the advantages of relational modeling over decoupling would strengthen the motivation and novelty positioning.

W5. Computational efficiency comparisons with direct baselines are missing. While the Appendix reports ULIFlow's inference time and VRAM usage across sequence lengths, comparisons with InterAct and LIGHT are absent. Reporting parameter counts, inference latency, and peak VRAM would help clarify the accuracy-efficiency trade-off.

Questions for authors

Please refer to the concerns raised in the weaknesses above.

7 comments
AuthorsOpenReview ↗

Response to Critical Weaknesses (Part 1)

We sincerely thank Reviewer PTqE for recognizing our “clear and well motivated” problem definition and the “meaningful improvements over prior baselines” demonstrated by our experiments. We address the questions and concerns below.

W1. The necessity of the explicit... Thank you very much for pointing out this gap in our ablation design. We have conducted three additional ablation experiments to disentangle the contributions of the relation pathway’s components. Specifically, we (i) retain relation reconstruction only as an auxiliary task without an explicit relation stream or relation-guided readout, (ii) remove the dedicated relation-guided readout for human and object decoding while retaining the relation stream and its supervision, and (iii) remove the relation reconstruction loss while retaining the relation stream and its readout.

In the following tables, we denote variants (i), (ii), and (iii) as w/o explicit relation stream, w/o R-guided readout, and w/o relation supervision, respectively. Full denotes the complete model, while w/o relation branch removes the relation stream, relation reconstruction supervision, and relation-guided readout.

To assess how each relation mechanism affects generative quality, we evaluate the five configurations on InterAct. FID measures feature-distribution similarity, while R-Precision and MM-Dist assess text–interaction alignment.

Generative Quality

ConfigurationFID ↓R@1 ↑R@2 ↑R@3 ↑MM-Dist ↓
Full0.812 ± 0.0140.852 ± 0.0080.945 ± 0.0080.972 ± 0.0054.852 ± 0.013
w/o relation branch1.244 ± 0.0260.861 ± 0.0100.947 ± 0.0070.976 ± 0.0014.890 ± 0.024
w/o explicit relation stream1.401 ± 0.0120.845 ± 0.0020.941 ± 0.0040.967 ± 0.0075.026 ± 0.014
w/o R-guided readout1.191 ± 0.0260.848 ± 0.0010.947 ± 0.0040.969 ± 0.0045.001 ± 0.004
w/o relation supervision1.118 ± 0.0230.852 ± 0.0020.947 ± 0.0080.968 ± 0.0074.995 ± 0.013

Full-Body Contact Quality

ConfigurationPrecision (%) ↑Recall (%) ↑F1 (%) ↑Contact dist. (cm) ↓
Full83.34 ± 0.0588.24 ± 0.2284.40 ± 0.192.819 ± 0.050
w/o relation branch79.34 ± 0.2076.83 ± 0.3774.79 ± 0.174.457 ± 0.035
w/o explicit relation stream79.66 ± 0.1376.94 ± 0.6475.36 ± 0.374.141 ± 0.123
w/o R-guided readout80.05 ± 0.1985.27 ± 0.2679.66 ± 0.183.213 ± 0.050
w/o relation supervision78.52 ± 0.1875.94 ± 0.4274.00 ± 0.144.212 ± 0.058

Hand Contact Quality

ConfigurationPrecision (%) ↑Recall (%) ↑F1 (%) ↑Contact dist. (cm) ↓
Full72.39 ± 0.3472.29 ± 0.2070.37 ± 0.204.323 ± 0.031
w/o relation branch68.59 ± 0.5153.25 ± 0.4455.95 ± 0.376.856 ± 0.104
w/o explicit relation stream68.80 ± 0.1456.40 ± 0.2658.25 ± 0.296.217 ± 0.046
w/o R-guided readout69.90 ± 0.4667.77 ± 0.5864.99 ± 0.364.861 ± 0.079
w/o relation supervision66.99 ± 0.1753.23 ± 0.5955.23 ± 0.386.766 ± 0.046

Although generative performance remains broadly comparable across configurations, the contact results reveal clear differences in human–object coordination. This indicates that individually plausible human and object motions do not by themselves ensure coordinated interaction, which motivates the introduction of an explicit relation branch.

Overall, all ablated configurations exhibit lower contact quality than Full, highlighting the contribution of each component to human–object coordination. Among these ablations, removing the relation-guided readout leads to the smallest decline in contact quality, suggesting that relation information already shapes the human and object latent representations during joint modeling and contributes to their coordination. Consistent with this observation, retaining relation reconstruction solely as an auxiliary task yields only limited contact improvements over removing the relation branch entirely. This further highlights the importance of allowing explicit relation latent tokens to participate in MoT’s shared attention, where information about continuously evolving relative geometry influences both the human and object streams and promotes coordination. When relation reconstruction supervision is removed, however, contact performance drops to a level comparable to removing the relation branch entirely. This indicates that, even when a relation stream participates in human and object modeling, explicit geometric supervision is essential for learning meaningful relational features that support coordination. Together, these results show that the relation branch’s coordination gains stem from the complementary contributions.

AuthorsOpenReview ↗

Response to Critical Weaknesses (Part 2)

W2. The methodological novelty... Our methodological contribution is coordination-oriented explicit relation modeling within a unified generative latent space. By learning continuous human–object relations alongside individual dynamics and incorporating them into coupled decoding, our formulation makes interaction geometry an explicit part of the representation used for generation. The innovation lies in learning and using this relational structure to coordinate human and object motion, with MoT and body-surface markers serving as established technical foundations.

As detailed in our response to Reviewer vwGW’s W2, parameter-matched autoencoder comparisons show that allocating capacity to the relation branch maintains comparable reconstruction accuracy and contact fidelity. Together with the component-wise generation and contact ablations reported in our W1 response, these results support a latent modeling design that improves coordination while preserving individual motion reconstruction. We will make this distinction between our coordination-oriented formulation and its established building blocks explicit in the contribution statements.

W3. Direct baseline comparisons... To assess contact quality beyond penetration, we have added direct comparisons with InterAct and LIGHT on the full InterAct evaluation set. We evaluate full-body and hand contact separately, reporting the mean and standard deviation across three seeds. Precision, recall, and F1 measure agreement with reference contact states, while contact distance measures the human–object gap during reference contact frames.

Full-Body Contact Quality

MethodPrecision (%) ↑Recall (%) ↑F1 (%) ↑Contact dist. (cm) ↓
InterAct77.02 ± 0.2278.26 ± 0.5874.70 ± 0.464.428 ± 0.079
LIGHT73.96 ± 0.4275.68 ± 0.4371.19 ± 0.344.782 ± 0.106
Ours83.34 ± 0.0588.24 ± 0.2284.40 ± 0.192.819 ± 0.050

Hand Contact Quality

MethodPrecision (%) ↑Recall (%) ↑F1 (%) ↑Contact dist. (cm) ↓
InterAct65.77 ± 0.0957.12 ± 0.4257.25 ± 0.246.668 ± 0.086
LIGHT58.15 ± 0.5844.87 ± 0.3046.16 ± 0.338.690 ± 0.150
Ours72.39 ± 0.3472.29 ± 0.2070.37 ± 0.204.323 ± 0.031

Across both evaluations, ULIFlow exhibits more reliable contact and smaller human–object gaps than both baselines. Together with the penetration results, these improvements show that reduced interpenetration is accompanied by better contact preservation, addressing the concern that lower penetration could arise from excessive separation. The joint improvement supports more coordinated interactions in which the human and object remain appropriately engaged while avoiding interpenetration.

AuthorsOpenReview ↗

Response to Critical Weaknesses (Part 3)

W4. The related work... DecHOI improves coordination through two complementary mechanisms. First, it separates global trajectory planning from detailed action synthesis, allowing human and object motions to follow jointly generated paths. Second, it employs adversarial training focused on distal joints and object geometry to encourage realistic contact.

However, DecHOI also applies explicit geometric corrections during sampling through reconstruction guidance. Specifically, contact masks constrain hand–object distances, while additional constraints regulate foot–floor contact. Consequently, its coordination mechanism remains dependent on predefined contact criteria and guidance hyperparameters, which is not able to generalize across diverse interaction scenarios and levels of granularity.

By comparison, our method explicitly models continuously evolving relations. During inference, these learned relations implicitly influence human and object generation and constrain their relative dynamics. In this way, our method improves coordination without relying on any hand-crafted designs.

W5. Computational efficiency... To clarify the accuracy–efficiency trade-off, we have added direct comparisons with InterAct and LIGHT on the same RTX 5080 GPU, using a batch size of one and 128 valid output frames. Inference latency is reported as the mean and standard deviation over five measured runs after two warm-up runs. Parameter counts include inference-resident networks, including frozen condition encoders and our AE decoder.

Inference Efficiency

MethodParameters (M) ↓Inference time (s) ↓Peak VRAM (GiB) ↓
InterAct119.6414.9366 ± 0.00160.476
LIGHT114.91314.2398 ± 0.00510.581
Ours208.3182.4260 ± 0.00291.108

Timing includes condition encoding, sampling, and native decoding, excluding model loading, representation conversion, and rendering. LIGHT retains its official guided sampler and 300-frame internal window, of which 128 frames are valid. Peak VRAM denotes peak allocated GPU memory.

Although ULIFlow uses more parameters and GPU memory, it achieves lower inference latency than both baselines. Together with the generation and contact results, these measurements show that our quality and coordination improvements are accompanied by faster generation, at the cost of a larger model and memory footprint.

Reviewer_vwGWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ULIFlow, which generates human-object interactions from text, object geometry, and an indicated duration. The main contribution of this paper is to use continuous marker-object relations as an explicit modality, by a Mixture-of-Transformers autoencoder. Additonal relation features are used in human and object decoding as well. A latent flow model on the latent space, uses blockwise Flow Forcing and sequence-progress conditioning to extend generation beyond fixed training windows. Experiments on standard benchmark InterAct, show the merit of the work with better generation quality, contact and penetration, and longer rollouts, with ablations of the main components.

Critical strengths

Explicitly modeling the geometric relationship of human and object is a well-motivated idea, thought not very novel, and the experiments provid evidence that it improves coordination, e.g., the ablation study shows the strength of the proposed component, where the relation-branch ablation showing the improvement over contact and object motion

Critical weaknesses
  1. I am not fully convinced by the evaluation of contact quality. The qualitative examples are too limited to show whether the proposed design choices consistently improve human–object contact. More visualizations and videos, including randomly selected examples and failure cases, would help assess how reliable these improvements are and reduce concerns about cherry-picking.

  2. The autoencoder is one of the major component as the contribution, but its validation could be more convincing. Although reconstruction errors are reported, it remains unclear how well the learned representation preserves interaction details such as contact and human-object alignment. I am also curious whether how the redundant representation in the reconstruction affect the performance More reconstruction visualizations, contact measurements before and after reconstruction, and evaluation on unseen objects or motions would help understand its fidelity and generalization.

  3. This is not critical but minor. I find the idea interesting, but its novelty over closely related work could be clearer. InterAct already uses marker-based representations, while LIGHT explores asynchronous denoising. And considering other related work in modeling the relation of human and object, the major novelty may lie in just architectural change

Questions for authors

It seems like LIGHT performs substantially worse than the earlier InterAct method in demo video. Could the authors explain this gap? While I am not requesting a comparison with concurrent work, the baseline examples in PAMI (https://arxiv.org/abs/2609.38466) appear more plausible. It would be helpful to clarify whether differences in implementation account for this discrepancy.

Somehow I feel the tokenization paragraph is difficult to follow. It would help to explain what each token represents, how the human, object, and relation features are converted into tokens, and how these tokens are organized across time. A simple example, diagram, or the formula showing the token structure would make the description clearer.

2 comments
AuthorsOpenReview ↗

Response to Critical Weaknesses

We thank the reviewer for the constructive feedback and for recognizing the benefits of explicit relation modeling.

Below, we address the concerns in the proposed critical weaknesses.

We have added a rebuttal folder to the supplementary material. It contains two subfolders, reconstruction_comparison and generative_results_comparison, which we refer to as F1 and F2, respectively, throughout this response.

W1. I am not fully convinced... To present more generated videos, we randomly selected ten different object types and corresponding text prompts for generative results comparison in F2. F2 presents comparisons between ours and ours w/o relation, showing differences in contact quality. Specifically, ours w/o relation often generates individually plausible human and object trajectories but exhibits noticeably poorer contact quality.

W2. The autoencoder is one of... To assess the effect of the redundant representation (additional relation representation), we compared autoencoders with and without the relation branch under a matched parameter budget, evaluating how allocating capacity to this branch affects contact quality after reconstruction.

Reconstruction errors

MethodMPMPE ↓ (cm)Hand MPMPE ↓ (cm)Object translation error ↓ (cm)Object rotation error ↓ (Frob.)
Ground Truth0.0000.0000.0000.000
Ours w/o relation1.2931.9311.5320.035
Ours1.3331.8061.4260.038

Contact quality

MethodBody Precision ↑ (%)Body Recall ↑ (%)Body F1 ↑ (%)Hand F1 ↑ (%)Contact distance ↓ (cm)
Ground Truth99.32799.32799.32790.2441.860
Ours w/o relation96.23195.61495.62685.6471.948
Ours96.14695.97495.69686.4371.894

Additionally, we randomly selected ten object–caption pairs covering different object types and included their reconstruction videos in F1. We can observe from both the tables and videos that allocating model capacity to the additional relation branch maintains comparable reconstruction performance. Taking the tables and the qualitative examples in F2 together, we can find that the additional relation branch maintains comparable reconstruction performance while helping the generative model capture interaction dynamics and improve human–object coordination.

W3. This is not critical but minor... We thank the reviewer for highlighting the need to clarify the differences between our method and prior work.

  • Regarding InterAct: InterAct derives marker–object offset vectors from its marker-based representation and uses the predicted relations to explicitly guide generation during sampling. In contrast, ULIFlow jointly models human and object dynamics in a relation-aware latent space, where relation features directly influence the generation of both trajectories through learned coupling. This formulation supports supervision beyond instantaneous spatial offsets, including relative velocity and acceleration.
  • Regarding LIGHT: LIGHT learns contact information using threshold-derived binary labels, which distinguish contact from non-contact without explicitly describing how relative geometry evolves. By comparison, we supervise continuous relative geometry and its temporal evolution, allowing the model to capture the continuous progression of approach, coordinated motion, and separation. Furthermore, LIGHT requires a separately configured, two-pass sampling procedure to obtain its pace-induced coordination gains. This introduces an additional inference-stage intervention for coordination. Our method instead embeds relational coupling into the learned latent representation and decoder, allowing human and object dynamics to be coordinated internally without auxiliary sampling-time guidance.

We thank the reviewer for highlighting the need for a clearer comparison with related work, which has helped us better articulate the novelty of our method. We will incorporate these clarifications into the revised paper.

AuthorsOpenReview ↗

Response to Questions For Authors

Here, we address the reviewer’s questions.

Q1. It seems like... Upon investigation, we found that the LIGHT results in our demo video were generated without guidance, resulting in some unusual behavior. However, all tabulated comparisons in the paper used the correct guidance configuration recommended by LIGHT’s official implementation. We have now regenerated and rendered these sequences using the officially recommended guidance configuration. The updated video, baseline_comparison.mp4, has been added to the rebuttal folder of the supplementary material for your review.

Q2. Somehow I feel... We thank the reviewer for this suggestion.

We clarify below what each token represents and how the tokens are organized across time.

To allow the model to adaptively emphasize different body regions as the interaction evolves, we use learned queries to aggregate marker-level features through cross-attention. Specifically, human motion features and marker–object relation features are projected into a common embedding space and aggregated by two separate sets of eight queries. Within each stream, the queries are reused across frames, while their attention weights adapt to the current motion and interaction geometry. Each resulting token therefore represents a learned, context-dependent summary without imposing a fixed anatomical partition.

At each frame, the three dynamic streams contain:

  • Human: eight query-aggregated motion tokens, one root-motion token, and two foot tokens encoding the corresponding foot-marker heights, yielding 11 tokens.
  • Object: one token encoding the rigid-object pose and its motion features.
  • Relation: eight query-aggregated local-relation tokens, one object–ground token, and one root–object token, yielding 10 tokens.

In addition, four learned queries aggregate the static BPS geometry into four geometry tokens, which are included once per sequence rather than repeated at every frame. Omitting the batch dimension, the token layout before flattening is

X^O_{\mathrm{pose}}\in\mathbb{R}^{T\times1\times d},\qquad X^R\in\mathbb{R}^{T\times10\times d},\qquad X^G\in\mathbb{R}^{4\times d},$$ where $T$ is the number of frames and $d$ is the common embedding width. To preserve token identities, token-type and slot embeddings distinguish their roles within each modality. Although the streams retain separate modality-specific transformations, their dynamic tokens use RoPE with shared temporal coordinates, so tokens corresponding to the same instant remain aligned across streams. Static geometry tokens retain their learned identities but receive no temporal rotation. Subsequent temporal compression maps all three dynamic streams onto shared temporal anchors while retaining their token identities. For example, a fully valid 128-frame window is compressed to 32 anchors, yielding $32 \times (11+1+10)+4=708$ latent tokens. This organization preserves both modality identity and a common temporal reference. We will revise the unclear descriptions in the final paper to improve readability, and we thank the reviewer for this helpful suggestion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.