- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HHI-Dyn, a framework for text-driven human-human interaction motion generation. The core contributions include: (1) STC-VAE, which constructs more robust latent representations by dynamically fusing global and local spatial information; (2) Inter-DiT, which introduces an Interactive Expert Module to capture different types of interaction patterns. Experiments are conducted on InterHuman and InterX datasets.
Critical strengths
- Human-human interaction motion generation is a practically meaningful task, and the paper's analysis of challenges (latent space construction difficulty, interaction complexity) is reasonable.
- The authors provide multiple sets of ablation experiments (Tables 3, 4, 5), verifying the effectiveness of causal encoding, interactive expert module, spatial fusion strategies, etc.
- A user preference study involving 20 users and 30 text prompts was conducted, adding credibility to the evaluation.
- On the InterHuman dataset, FID reaches 4.658 and R-Precision Top-1 reaches 0.515, showing some improvement over baselines.
Critical weaknesses
-
The "global-local spatial fusion" in STC-VAE is essentially adding a gated fusion mechanism to existing VAE architectures. Similar ideas have been extensively explored in image generation (e.g., StyleGAN series) and multimodal learning. The Interactive Expert Module is fundamentally a direct application of Mixture-of-Experts (MoE) architecture to motion generation, combined with variants of cross-attention mechanisms. The overall framework is a standard "VAE + Latent Diffusion + Transformer" pipeline, where each component is a combination of existing techniques.
-
In Table 2, on the InterX dataset, HHI-Dyn's FID (0.314) underperforms InterMoE (0.297), yet the authors only emphasize the "second-best" result in the text without adequate explanation. Table 1 claims to show advantages in "Dynamic Global Local Fusion" and "Capturing Interaction Types" over InterMoE, but the FID difference (4.658 vs 4.677) is merely 0.019, and R-Precision (0.515 vs 0.512) shows marginal improvement. These differences may not be statistically significant.
-
The paper claims to be the "first" to perform global-local spatial fusion, but prior works (e.g., part-wise motion generation) have made similar attempts. The authors should more accurately position their contribution.
-
The conclusion only briefly discusses "physical plausibility" and "multi-person interaction" in one paragraph, without addressing more critical issues such as generalization to unseen interaction types, quality degradation in long sequence generation, and training efficiency.
Questions for authors
- On the InterX dataset, HHI-Dyn's FID (0.314) is worse than InterMoE (0.297). Please explain why. Does this indicate that your model's generalization ability across different skeleton representations is still limited?
- Training is conducted on a single RTX 3090. Is this configuration sufficient for diffusion models that typically require substantial computational resources? Could there be under-optimization due to computational constraints?
- While HHI-Dyn generates visually more natural poses, there is no quantitative metric to evaluate "pose robustness." The advantage claimed on page 7 lacks objective evidence.
5 comments
Author Rebuttal (2/2)
Question 4: Why does HHI-Dyn achieve a slightly worse FID than InterMoE on InterX? Does this indicate limited generalization?
Thank you for this important question! We acknowledge that HHI-Dyn achieves a slightly higher FID than InterMoE on InterX (0.314 vs. 0.297). However, HHI-Dyn outperforms InterMoE in R-Precision Top-1, Top-2, Top-3, and MM-Dist, demonstrating stronger text–motion semantic alignment. This suggests that our framework remains effective on InterX despite the modest FID difference.
Importantly, InterHuman and InterX employ different skeletal representations, with InterX involving more complex SMPL-X motion features. Our framework can accommodate both representations without modifying its core modeling principles, demonstrating its applicability across different motion representations. Nevertheless, we acknowledge that the FID difference indicates room for further improvement in modeling complex skeletal structures.
Question 5: Is training on a single RTX 3090 sufficient? Could computational constraints lead to under-optimization?
Thank you for raising this important concern! We believe that training on a single RTX 3090 highlights the computational practicality of our framework rather than indicating under-optimization. In contrast to methods such as TiMotion, which employ 8 NVIDIA L40S GPUs, HHI-Dyn can be trained using only a single RTX 3090. This is partly enabled by performing diffusion in a compressed latent space rather than directly in the high-dimensional motion space, substantially reducing computational demands.
Furthermore, our competitive results on both InterHuman and InterX, together with extensive ablation studies, demonstrate the effectiveness of our training configuration. While additional computational resources may offer further optimization opportunities, our results show that HHI-Dyn achieves competitive generation quality with relatively modest hardware requirements.
Question 6: How is the claimed improvement in pose robustness objectively validated?
Thank you for this valuable comment! We evaluate the quality and naturalness of generated motions through both quantitative and qualitative experiments. Quantitatively, we adopt widely used metrics, including FID and R-Precision, and achieve competitive results in human-human interaction generation, as shown in Table 2.
Qualitatively, we provide extensive visual comparisons in Figures 3 and 4, demonstrating that our generated motion sequences exhibit more natural poses and better alignment with textual descriptions. Furthermore, our user study provides additional perceptual evidence supporting the quality and naturalness of the generated interactions.
Official Comment by Authors
We sincerely appreciate the time you took to review our manuscript and provide insightful comments. We hope our point-by-point response to your feedback addresses your concerns. We are writing to respectfully inquire whether our responses have successfully resolved all your questions and concerns. Please let us know if there is any additional information you require from our side.
Thank you very much!
Author Rebuttal (1/2)
We sincerely thank the reviewer for the constructive feedback and recognition of our ablation studies, user study, and improvements on InterHuman. We address your concerns regarding performance, training resources, pose robustness, and methodological novelty below.
Question 1: What is the methodological novelty of STC-VAE and the Interactive Expert Module beyond existing gated fusion and MoE architectures?
Thank you for this insightful comment! While prior part-wise motion generation methods primarily focus on partitioning the human body into separate skeletal regions, our STC-VAE goes beyond spatial partitioning by dynamically integrating each local body part with global motion information. To the best of our knowledge, we are among the first to introduce such adaptive part-wise global–local fusion for human-human motion generation.
Specifically, STC-VAE introduces dynamic global–local spatial fusion, where each skeletal body part adaptively integrates global body motion information during reconstruction. Unlike conventional global-only or part-wise representations, this design explicitly couples each local skeletal region with the overall body context, producing richer latent representations for subsequent interaction generation.
Furthermore, our Interactive Expert Module extends conventional MoE routing to cooperative interaction-aware expert selection. Rather than relying solely on individual motion and textual semantics, our Cooperative Router jointly incorporates individual motion, text, and partner motion, with cosine similarity further modulating partner-dependent routing. To the best of our knowledge, this specific cooperative routing formulation has not been explored in prior text-driven two-person interaction generation methods.
Importantly, our ablation studies validate both designs: global–local fusion reduces FID from 5.898 (global-only) and 4.814 (local-only) to 4.658, while removing the Partner Router increases FID from 4.658 to 5.146. These results support the effectiveness of our proposed interaction-specific designs beyond simply combining existing components.
Question 2: How significant are the improvements over InterMoE, given the marginal quantitative differences and the slightly worse FID on InterX?
Thank you for this important comment! First, HHI-Dyn achieves the best FID (4.658) on InterHuman, outperforming InterMoE (4.677). Although our FID on InterX is slightly worse than InterMoE (0.314 vs. 0.297), it remains the second-best reported result. More importantly, HHI-Dyn outperforms InterMoE in text–motion alignment metrics, including R-Precision Top-1, Top-2, Top-3, and MM-Dist on InterX, which are particularly important for text-driven motion generation.
Furthermore, our improvements extend beyond quantitative metrics. Qualitative comparisons demonstrate that HHI-Dyn generates interaction motions with better alignment to textual descriptions than InterMoE. Additionally, our user study provides perceptual evidence of improved motion fidelity and text–motion consistency, capturing aspects of generation quality that may not be fully reflected by FID alone. Detailed visual comparisons and demonstrations are provided in the supplementary material.
User Study Results:
The combined quantitative, qualitative, and user-study results support the effectiveness of our framework.
Question 3: How does HHI-Dyn address limitations regarding generalization to unseen interaction types, long-sequence generation, and training efficiency?
Thank you for this valuable suggestion! In our conclusion, we have discussed physical plausibility and multi-person interaction generation as important directions for future research.
Regarding unseen interaction types and long-sequence generation, current datasets provide limited coverage of diverse interactions and long-duration motion sequences, making systematic evaluation challenging. Nevertheless, we provide additional long-sequence generation examples in the supplementary material, demonstrating the potential of HHI-Dyn to generate extended interaction motions. We acknowledge that generalization to unseen interactions and temporal consistency over longer sequences require further investigation.
Regarding training efficiency, HHI-Dyn can be trained on a single NVIDIA RTX 3090 GPU, with the diffusion stage requiring approximately two days of training. This demonstrates the computational practicality of our framework without relying on extensive GPU resources.