- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ULIFlow, a framework for text-conditioned 3D human–object interaction (HOI) generation with variable length. The authors derive continuous marker–object relations (e.g., relative offsets, distances, and their temporal differences) from body-surface markers and treat them as an additional modality alongside human and object motion. A Mixture-of-Transformers (MoT) autoencoder jointly encodes the three modalities into a unified latent space, and a conditional latent DiT is trained with a blockwise Flow Forcing scheme and sequence-progress conditioning to generate HOI sequences beyond the fixed training window. Experiments on the InterAct dataset show improvements over prior methods in generation quality and human–object penetration.
Critical strengths
- This paper introduces a new marker-based representation that explicitly models interaction geometry, which has not been explored in previous methods. Specifically, the authors explicitly model continuous human–object relations within a unified latent space and introduce a Mixture-of-Transformers autoencoder that jointly encodes human, object, and relation modalities.
Critical weaknesses
- The comparison in Table 1 is unfair, and some results are questionable.
- Several compared methods are not trained on InterAct (e.g., CHOIS and HOI-Diff are trained on OMOMO and BEHAVE, respectively), which leads to an out-of-distribution (OOD) evaluation, whereas the proposed method is trained and tested on InterAct.
- The reproduced results seem unusual. For example, as shown in Table 1, HOI-Diff achieves very poor performance (0.134 Top-1 R-Precision and 14.445 FID), which is much worse than CHOIS. In contrast, the results reported in the LIGHT paper show only a small performance gap between HOI-Diff and CHOIS.
- The authors claim that their method supports variable-length generation, but they do not provide sufficient experiments to support this claim.
- All baselines in Table 2 and Fig. 3 are single-window methods, and it is unclear how their long sequences and "continuation boundaries" are obtained. A continuation baseline (e.g., InterAct with sliding-window inpainting) is needed for a fair comparison.
- In the ablations, the variant without blockwise forcing achieves better FID than the full model in several settings (Table C, 128 frames: 6.837 vs. 7.712; Table D: 4.853 vs. 4.917 and 3.554 vs. 3.800), so the benefit of the proposed schedule is mainly reflected in jerk statistics.
- All models are trained only on the InterAct dataset. Although Table B reports results on the BEHAVE and OMOMO subsets, these are subsets of InterAct evaluated with the model trained on the full InterAct dataset. It would be better if the authors also trained and evaluated their method on the original OMOMO and BEHAVE benchmarks following their standard protocols, so that the results can be directly compared with the numbers reported in prior work.
Questions for authors
See Weaknesses.
5 comments
Response to Critical Weaknesses (Part 2)
W2. The authors claim that...
- Regarding the long-sequence performance comparison: To address this concern, we extend InterAct with sliding-window inpainting to construct a continuation baseline. Specifically, each new 128-frame window is conditioned on the final 32 generated frames of the preceding window. For a matched comparison, we evaluate both methods on long sequences from the dataset, with the generation length set to 384 frames. Across the rollout, we evaluate non-overlapping 128-frame segments from the early, middle, and late stages, and we further examine 64-frame neighborhoods centered on InterAct’s actual continuation boundaries at positions 128, 224, and 320.
Early stage (frames 1–128)
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| InterAct | 9.786 ± 0.612 | 0.656 ± 0.028 | 6.239 ± 0.030 | 69.38 ± 1.34 | 47.55 ± 1.48 |
| Ours | 5.645 ± 0.175 | 0.667 ± 0.038 | 6.424 ± 0.038 | 74.95 ± 1.19 | 57.32 ± 1.78 |
Middle stage (frames 129–256)
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| InterAct | 10.046 ± 1.154 | 0.587 ± 0.047 | 6.738 ± 0.144 | 70.15 ± 2.26 | 46.03 ± 0.75 |
| Ours | 5.905 ± 0.152 | 0.667 ± 0.028 | 6.323 ± 0.008 | 85.87 ± 0.39 | 66.23 ± 1.17 |
Late stage (frames 257–384)
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| InterAct | 9.862 ± 0.953 | 0.569 ± 0.033 | 7.108 ± 0.035 | 70.27 ± 2.39 | 45.03 ± 1.61 |
| Ours | 7.245 ± 0.030 | 0.660 ± 0.024 | 6.539 ± 0.031 | 80.95 ± 0.26 | 62.75 ± 1.22 |
Boundary at position 128
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| InterAct | 9.566 ± 1.979 | 0.538 ± 0.030 | 6.971 ± 0.097 | 67.62 ± 1.22 | 44.57 ± 1.05 |
| Ours | 4.523 ± 0.121 | 0.653 ± 0.037 | 6.483 ± 0.032 | 80.94 ± 1.05 | 61.31 ± 0.84 |
Boundary at position 224
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| InterAct | 8.368 ± 1.007 | 0.566 ± 0.012 | 6.920 ± 0.140 | 67.64 ± 2.29 | 42.08 ± 1.39 |
| Ours | 5.323 ± 0.166 | 0.608 ± 0.037 | 6.680 ± 0.037 | 84.76 ± 0.69 | 65.33 ± 0.98 |
Boundary at position 320
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| InterAct | 10.510 ± 1.466 | 0.528 ± 0.043 | 7.251 ± 0.056 | 65.66 ± 3.82 | 40.72 ± 3.40 |
| Ours | 6.003 ± 0.117 | 0.604 ± 0.048 | 6.848 ± 0.097 | 80.91 ± 0.43 | 63.18 ± 0.23 |
Here, FID, R@1, and MM-Dist assess generation quality, while Body F1 and Hand F1 assess full-body and hand–object contact quality, respectively.
Across all evaluated settings, our method exhibits a clear overall advantage over InterAct with sliding-window inpainting. These results demonstrate the effectiveness of our design for long-horizon generation.
Response to Critical Weaknesses (Part 3)
- Regarding the effectiveness of temporal modeling designs: To clarify how our temporal modeling designs affect long-horizon generation, we compare the full model with variants without blockwise forcing and without sequence-progress conditioning. Specifically, we generate 384-frame rollouts using long sequences from the dataset and separately evaluate their early, middle, and late 128-frame segments. We also evaluate 64-frame neighborhoods centered at positions 128, 224, and 320, matching the continuation-boundary evaluation above.
Early stage (frames 1–128)
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| Ours w/o progress | 7.358 ± 0.225 | 0.674 ± 0.016 | 6.168 ± 0.034 | 70.73 ± 0.21 | 53.47 ± 0.85 |
| Ours w/o blockwise | 5.680 ± 0.066 | 0.674 ± 0.016 | 6.394 ± 0.008 | 77.06 ± 0.95 | 59.18 ± 1.58 |
| Ours (Full) | 5.645 ± 0.175 | 0.667 ± 0.038 | 6.424 ± 0.038 | 74.95 ± 1.19 | 57.32 ± 1.78 |
Middle stage (frames 129–256)
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| Ours w/o progress | 7.198 ± 0.307 | 0.587 ± 0.053 | 6.728 ± 0.073 | 69.02 ± 0.89 | 51.49 ± 2.47 |
| Ours w/o blockwise | 6.018 ± 0.019 | 0.646 ± 0.036 | 6.373 ± 0.028 | 84.96 ± 0.54 | 65.86 ± 0.53 |
| Ours (Full) | 5.905 ± 0.152 | 0.667 ± 0.028 | 6.323 ± 0.008 | 85.87 ± 0.39 | 66.23 ± 1.17 |
Late stage (frames 257–384)
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| Ours w/o progress | 7.537 ± 0.064 | 0.632 ± 0.026 | 6.684 ± 0.059 | 74.20 ± 1.24 | 55.55 ± 1.32 |
| Ours w/o blockwise | 7.665 ± 0.054 | 0.628 ± 0.006 | 6.583 ± 0.046 | 79.65 ± 1.25 | 61.42 ± 1.42 |
| Ours (Full) | 7.245 ± 0.030 | 0.660 ± 0.024 | 6.539 ± 0.031 | 80.95 ± 0.26 | 62.75 ± 1.22 |
Boundary at position 128
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| Ours w/o progress | 5.938 ± 0.136 | 0.562 ± 0.031 | 6.749 ± 0.011 | 68.30 ± 0.53 | 50.66 ± 0.41 |
| Ours w/o blockwise | 4.758 ± 0.141 | 0.705 ± 0.033 | 6.368 ± 0.010 | 82.99 ± 0.24 | 63.43 ± 2.65 |
| Ours (Full) | 4.523 ± 0.121 | 0.653 ± 0.037 | 6.483 ± 0.032 | 80.94 ± 1.05 | 61.31 ± 0.84 |
Boundary at position 224
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| Ours w/o progress | 6.054 ± 0.276 | 0.559 ± 0.043 | 6.827 ± 0.057 | 68.68 ± 1.76 | 51.17 ± 3.17 |
| Ours w/o blockwise | 5.479 ± 0.148 | 0.608 ± 0.026 | 6.692 ± 0.038 | 83.83 ± 0.59 | 65.04 ± 1.12 |
| Ours (Full) | 5.323 ± 0.166 | 0.608 ± 0.037 | 6.680 ± 0.037 | 84.76 ± 0.69 | 65.33 ± 0.98 |
Boundary at position 320
| Method | FID ↓ | R@1 ↑ | MM-Dist ↓ | Body F1 (%) ↑ | Hand F1 (%) ↑ |
|---|---|---|---|---|---|
| Ours w/o progress | 6.520 ± 0.311 | 0.583 ± 0.073 | 6.868 ± 0.054 | 72.79 ± 1.67 | 54.59 ± 0.38 |
| Ours w/o blockwise | 6.245 ± 0.062 | 0.576 ± 0.012 | 6.862 ± 0.077 | 78.90 ± 1.26 | 61.35 ± 1.95 |
| Ours (Full) | 6.003 ± 0.117 | 0.604 ± 0.048 | 6.848 ± 0.097 | 80.91 ± 0.43 | 63.18 ± 0.23 |
As shown in the tables, within frames 1–128, which match the training-window length, our temporal modeling designs introduce minor performance trade-offs. As generation extends beyond this window, however, these designs consistently improve both generation quality and human–object coordination. These results demonstrate the effectiveness of our temporal modeling designs for generation beyond the training-window length.
Response to Critical Weaknesses (Part 1)
We sincerely thank Reviewer y9xG for recognizing the value of explicitly modeling continuous human–object relations within a unified latent space.
We address the concerns and questions below.
W1. The comparison...
Comparison on BEHAVE
Comparison on OMOMO
As shown in the tables, training on the full InterAct dataset provides clear benefits compared with training only on OMOMO and BEHAVE. Nevertheless, even when trained only on OMOMO and BEHAVE, our method still achieves better overall performance than other methods trained on the full InterAct dataset.