acceptodds
Under review as a conference paper at ICLR 2027

HHI-Dyn: Learning Interaction Dynamics for Human-Human Motion Diffusion

Abstract

Text-driven human interaction generation demands a robust latent space structure, where each individual's motion patterns are jointly shaped by the textual description and the partner's behavior. Existing interaction models neglect the coupling between local joints and global motion representations. Moreover, they typically incorporate partner information solely through cross-attention module, failing to account for the varying influence of the partner across different interaction types. To address these limitations, we propose HHI-Dyn, a novel latent diffusion framework for human-human motion generation. HHI-Dyn first introduces a tailored VAE that effectively fuses global spatial features with local spatial details. At each time step, every local body part adaptively integrates with the global spatial context, yielding a more stable latent representation for subsequent diffusion. A latent diffusion model is then trained to capture this spatial structure conditioned on interaction text. Crucially, we design an interactive expert module for motion pattern learning, where each expert is responsible for capturing a specific type of motion pattern. These experts dynamically adapt to diverse input conditions and partner behaviors, effectively improving both the interactive quality and perceptual realism of the generated two-person interactions. Extensive experiments demonstrate that HHI-Dyn achieves state-of-the-art performance on InterHuman and InterX datasets, with significant improvements in text-motion alignment and motion realism. We will release code and models to facilitate reproducibility.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Interaction Generation, Latent Diffusion Model, Text-driven Motion Synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_C7ufOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HHI-Dyn, a framework for text-driven human-human interaction motion generation. The core contributions include: (1) STC-VAE, which constructs more robust latent representations by dynamically fusing global and local spatial information; (2) Inter-DiT, which introduces an Interactive Expert Module to capture different types of interaction patterns. Experiments are conducted on InterHuman and InterX datasets.

Critical strengths
  1. Human-human interaction motion generation is a practically meaningful task, and the paper's analysis of challenges (latent space construction difficulty, interaction complexity) is reasonable.
  2. The authors provide multiple sets of ablation experiments (Tables 3, 4, 5), verifying the effectiveness of causal encoding, interactive expert module, spatial fusion strategies, etc.
  3. A user preference study involving 20 users and 30 text prompts was conducted, adding credibility to the evaluation.
  4. On the InterHuman dataset, FID reaches 4.658 and R-Precision Top-1 reaches 0.515, showing some improvement over baselines.
Critical weaknesses
  1. The "global-local spatial fusion" in STC-VAE is essentially adding a gated fusion mechanism to existing VAE architectures. Similar ideas have been extensively explored in image generation (e.g., StyleGAN series) and multimodal learning. The Interactive Expert Module is fundamentally a direct application of Mixture-of-Experts (MoE) architecture to motion generation, combined with variants of cross-attention mechanisms. The overall framework is a standard "VAE + Latent Diffusion + Transformer" pipeline, where each component is a combination of existing techniques.

  2. In Table 2, on the InterX dataset, HHI-Dyn's FID (0.314) underperforms​ InterMoE (0.297), yet the authors only emphasize the "second-best" result in the text without adequate explanation. Table 1 claims to show advantages in "Dynamic Global Local Fusion" and "Capturing Interaction Types" over InterMoE, but the FID difference (4.658 vs 4.677) is merely 0.019, and R-Precision (0.515 vs 0.512) shows marginal improvement. These differences may not be statistically significant.

  3. The paper claims to be the "first" to perform global-local spatial fusion, but prior works (e.g., part-wise motion generation) have made similar attempts. The authors should more accurately position their contribution.

  4. The conclusion only briefly discusses "physical plausibility" and "multi-person interaction" in one paragraph, without addressing more critical issues such as generalization to unseen interaction types, quality degradation in long sequence generation, and training efficiency.

Questions for authors
  1. On the InterX dataset, HHI-Dyn's FID (0.314) is worse than InterMoE (0.297). Please explain why. Does this indicate that your model's generalization ability across different skeleton representations is still limited?
  2. Training is conducted on a single RTX 3090. Is this configuration sufficient for diffusion models that typically require substantial computational resources? Could there be under-optimization due to computational constraints?
  3. While HHI-Dyn generates visually more natural poses, there is no quantitative metric to evaluate "pose robustness." The advantage claimed on page 7 lacks objective evidence.
5 comments
AuthorsOpenReview ↗

Author Rebuttal (1/2)

We sincerely thank the reviewer for the constructive feedback and recognition of our ablation studies, user study, and improvements on InterHuman. We address your concerns regarding performance, training resources, pose robustness, and methodological novelty below.


Question 1: What is the methodological novelty of STC-VAE and the Interactive Expert Module beyond existing gated fusion and MoE architectures?

Thank you for this insightful comment! While prior part-wise motion generation methods primarily focus on partitioning the human body into separate skeletal regions, our STC-VAE goes beyond spatial partitioning by dynamically integrating each local body part with global motion information. To the best of our knowledge, we are among the first to introduce such adaptive part-wise global–local fusion for human-human motion generation.

Specifically, STC-VAE introduces dynamic global–local spatial fusion, where each skeletal body part adaptively integrates global body motion information during reconstruction. Unlike conventional global-only or part-wise representations, this design explicitly couples each local skeletal region with the overall body context, producing richer latent representations for subsequent interaction generation.

Furthermore, our Interactive Expert Module extends conventional MoE routing to cooperative interaction-aware expert selection. Rather than relying solely on individual motion and textual semantics, our Cooperative Router jointly incorporates individual motion, text, and partner motion, with cosine similarity further modulating partner-dependent routing. To the best of our knowledge, this specific cooperative routing formulation has not been explored in prior text-driven two-person interaction generation methods.

Importantly, our ablation studies validate both designs: global–local fusion reduces FID from 5.898 (global-only) and 4.814 (local-only) to 4.658, while removing the Partner Router increases FID from 4.658 to 5.146. These results support the effectiveness of our proposed interaction-specific designs beyond simply combining existing components.


Question 2: How significant are the improvements over InterMoE, given the marginal quantitative differences and the slightly worse FID on InterX?

Thank you for this important comment! First, HHI-Dyn achieves the best FID (4.658) on InterHuman, outperforming InterMoE (4.677). Although our FID on InterX is slightly worse than InterMoE (0.314 vs. 0.297), it remains the second-best reported result. More importantly, HHI-Dyn outperforms InterMoE in text–motion alignment metrics, including R-Precision Top-1, Top-2, Top-3, and MM-Dist on InterX, which are particularly important for text-driven motion generation.

Furthermore, our improvements extend beyond quantitative metrics. Qualitative comparisons demonstrate that HHI-Dyn generates interaction motions with better alignment to textual descriptions than InterMoE. Additionally, our user study provides perceptual evidence of improved motion fidelity and text–motion consistency, capturing aspects of generation quality that may not be fully reflected by FID alone. Detailed visual comparisons and demonstrations are provided in the supplementary material.

User Study Results:

MethodInteraction Quality ↑Adherence to Text ↑
InterMask3.233.10
HiTMM3.533.46
InterMoE3.633.56
HHI-Dyn4.064.30

The combined quantitative, qualitative, and user-study results support the effectiveness of our framework.


Question 3: How does HHI-Dyn address limitations regarding generalization to unseen interaction types, long-sequence generation, and training efficiency?

Thank you for this valuable suggestion! In our conclusion, we have discussed physical plausibility and multi-person interaction generation as important directions for future research.

Regarding unseen interaction types and long-sequence generation, current datasets provide limited coverage of diverse interactions and long-duration motion sequences, making systematic evaluation challenging. Nevertheless, we provide additional long-sequence generation examples in the supplementary material, demonstrating the potential of HHI-Dyn to generate extended interaction motions. We acknowledge that generalization to unseen interactions and temporal consistency over longer sequences require further investigation.

Regarding training efficiency, HHI-Dyn can be trained on a single NVIDIA RTX 3090 GPU, with the diffusion stage requiring approximately two days of training. This demonstrates the computational practicality of our framework without relying on extensive GPU resources.


AuthorsOpenReview ↗

Author Rebuttal (2/2)

Question 4: Why does HHI-Dyn achieve a slightly worse FID than InterMoE on InterX? Does this indicate limited generalization?

Thank you for this important question! We acknowledge that HHI-Dyn achieves a slightly higher FID than InterMoE on InterX (0.314 vs. 0.297). However, HHI-Dyn outperforms InterMoE in R-Precision Top-1, Top-2, Top-3, and MM-Dist, demonstrating stronger text–motion semantic alignment. This suggests that our framework remains effective on InterX despite the modest FID difference.

Importantly, InterHuman and InterX employ different skeletal representations, with InterX involving more complex SMPL-X motion features. Our framework can accommodate both representations without modifying its core modeling principles, demonstrating its applicability across different motion representations. Nevertheless, we acknowledge that the FID difference indicates room for further improvement in modeling complex skeletal structures.


Question 5: Is training on a single RTX 3090 sufficient? Could computational constraints lead to under-optimization?

Thank you for raising this important concern! We believe that training on a single RTX 3090 highlights the computational practicality of our framework rather than indicating under-optimization. In contrast to methods such as TiMotion, which employ 8 NVIDIA L40S GPUs, HHI-Dyn can be trained using only a single RTX 3090. This is partly enabled by performing diffusion in a compressed latent space rather than directly in the high-dimensional motion space, substantially reducing computational demands.

Furthermore, our competitive results on both InterHuman and InterX, together with extensive ablation studies, demonstrate the effectiveness of our training configuration. While additional computational resources may offer further optimization opportunities, our results show that HHI-Dyn achieves competitive generation quality with relatively modest hardware requirements.


Question 6: How is the claimed improvement in pose robustness objectively validated?

Thank you for this valuable comment! We evaluate the quality and naturalness of generated motions through both quantitative and qualitative experiments. Quantitatively, we adopt widely used metrics, including FID and R-Precision, and achieve competitive results in human-human interaction generation, as shown in Table 2.

Qualitatively, we provide extensive visual comparisons in Figures 3 and 4, demonstrating that our generated motion sequences exhibit more natural poses and better alignment with textual descriptions. Furthermore, our user study provides additional perceptual evidence supporting the quality and naturalness of the generated interactions.

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely appreciate the time you took to review our manuscript and provide insightful comments. We hope our point-by-point response to your feedback addresses your concerns. We are writing to respectfully inquire whether our responses have successfully resolved all your questions and concerns. Please let us know if there is any additional information you require from our side.

Thank you very much!

Reviewer_nhuvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes HHI-Dyn for text-conditioned human-human interaction generation. The framework consists of two major components. First, the authors introduce a Spatio-Temporal Hybrid Causal VAE (STC-VAE), which extracts both global spatial features and local body-part features and adaptively fuses them through a learned gating mechanism. Second, an Interactive Diffusion Transformer is proposed, where an expert-based module routes motion features according to the subject motion, partner motion, and text condition. The model is evaluated on InterHuman and InterX and achieves competitive results against recent HHI generation methods.

The paper is generally easy to follow and the proposed framework obtains reasonable quantitative and qualitative results. However, I have major concerns regarding the novelty positioning, especially with respect to recent structured latent HHI generation methods and expert-based interaction models.

Critical strengths
  1. The paper studies an important problem of text-conditioned human-human interaction generation and considers both motion representation learning and interaction modeling.

  2. The global-local spatial fusion is intuitive. The ablation results suggest that combining global and local representations performs better than using either one independently.

  3. The partner-aware routing design is reasonable for HHI, since different interaction types naturally require different levels of dependency between two people.

  4. The authors provide experiments on both InterHuman and InterX, together with ablations and user studies.

Critical weaknesses

1. Insufficient novelty positioning against DHVAE

The proposed STC-VAE is motivated by the need for a more structured latent representation for HHI generation. However, DHVAE[1] has already explored structured/disentangled latent modeling for human-human interaction generation by separating individual motion factors and interaction-level representations. Although HHI-Dyn uses a different decomposition axis, namely global versus local spatial representations, the high-level motivation and framework are closely related. The lack of discussion and citation of DHVAE makes the novelty claim incomplete. The authors should clearly explain how STC-VAE differs conceptually and technically from DHVAE.

2. The Interactive Expert Module is closely related to InterMoE [2], making the overall contribution appear incremental

InterMoE already introduces expert-based modeling for human interaction generation. HHI-Dyn mainly extends this idea with motion, text, and partner routers together with cosine-similarity-based modulation. Since the paper’s two main contributions are respectively close to structured latent modeling and MoE-based interaction modeling, the authors should better justify what fundamentally new insight is introduced by combining them.

3. The proposed global-local fusion may be too simple to support the claimed novelty.

The core STC-VAE fusion is essentially an adaptive interpolation between local and broadcast global features using a learned sigmoid weight map. While the ablation confirms that global-local fusion is better than global-only or local-only variants, stronger baselines such as concatenation, additive fusion, attention-based fusion, or parameter-matched alternatives are missing. Therefore, it remains unclear whether the gain comes from the proposed formulation itself or simply from increased representational capacity.

4. The improvement over InterMoE is marginal, while the paper makes relatively strong SOTA claims

On InterHuman, HHI-Dyn only improves FID from 4.677 to 4.658 and Top-1 R-Precision from 0.512 to 0.515, while Top-2 and Top-3 are slightly worse. On InterX, its FID is worse than InterMoE, 0.314 versus 0.297. Therefore, claims such as “substantial improvements” should be moderated, and efficiency/parameter comparisons are necessary to justify the additional complexity.

5. Several important design choices are insufficiently justified.

First, cosine similarity between two motion features is used as a proxy for interaction strength, but highly interactive motions can be complementary rather than similar, while similar motions can also be independent. Second, causal encoding is emphasized even though the task is offline sequence generation. The paper should explain why causal encoding is preferable to bidirectional encoding and provide a more controlled ablation. HHI-Dyn does not claim MeanFlow or one-step generation; in fact, it uses 50 DDIM sampling steps at inference, so the relationship to ARMFlow is mainly limited to causal temporal modeling rather than the generation paradigm

In addition, I have a few minor concerns that are not critical but may help improve the paper:

  1. The paper lacks explicit physical plausibility metrics, such as penetration, contact consistency, or inter-person collision measures.
  2. The claim of “reduced implicit bias” in the qualitative comparison is unclear and not supported by a formal definition or metric.
  3. The paper should report model size, FLOPs, training cost, and inference speed, especially because the method introduces dual encoders, fusion modules, multiple experts, and several routers.

Overall, I have a positive view of this work, and I would encourage the authors to address these concerns during the rebuttal, and I intend to increase my score.

[1] Zichen Geng, Zeeshan Hayder, Bo Miao, Jian Liu, Wei Liu, and Ajmal Mian. Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation. International Conference on Learning Representations (ICLR), 2026. [2] Lipeng Wang, Hongxing Fan, Haohua Chen, Zehuan Huang, and Lu Sheng. InterMoE: Individual-Specific 3D Human Interaction Generation via Dynamic Temporal-Selective MoE. Proceedings of the AAAI Conference on Artificial Intelligence, 40(3):2137–2145, 2026.

Questions for authors

Please see the major concern above,

My main concern is how it differs from the DHVAE, and I hope the author could explain more about the performacne gain.

8 comments
AuthorsOpenReview ↗

Author Rebuttal (1/3)

We sincerely thank the reviewer for the constructive feedback and recognition of our method's strengths. We appreciate the opportunity to clarify the differences from DHVAE and InterMoE, as well as the motivation and effectiveness of our designs.

Question 1: What are the key differences between STC-VAE and DHVAE?

Thank you for this important question! Although both methods explore structured latent representations, they differ fundamentally in their decomposition strategies and interaction modeling mechanisms.

  • DHVAE disentangles two-person motions into three latent variables: two individual motion representations and a shared interaction representation. Unlike our approach, its reconstruction stage does not explicitly partition skeletal structures or dynamically fuse global body posture with local joint dynamics. During generation, DHVAE primarily models the text-conditioned shared interaction latent, rather than explicitly performing partner-aware expert routing between the two individual motion representations during diffusion.

  • Our HHI-Dyn employs STC-VAE to independently encode each person's motion through dynamic global–local spatial fusion, preserving both overall body posture and fine-grained skeletal dynamics without introducing a shared interaction latent. In the subsequent diffusion stage, Inter-DiT jointly generates two motion representations, allowing each individual's motion to be influenced by textual semantics and partner dynamics. Furthermore, cooperative expert routing explicitly incorporates partner information into expert selection, enhancing interaction modeling.

Thus, DHVAE emphasizes disentangling individual and shared interaction representations, whereas HHI-Dyn emphasizes global–local spatial fusion and explicit partner-aware interaction modeling during diffusion.

Furthermore, HHI-Dyn outperforms DHVAE on InterHuman in both FID (4.658 vs. 5.015) and R-Precision Top-1 (0.515 vs. 0.496), supporting the effectiveness of our overall framework. We have included DHVAE in the related work and clarified these distinctions.

MethodR-Precision Top-1 ↑FID ↓MM-Dist ↓
DHVAE0.4965.0153.772
Ours0.5154.6583.765

Question 2: What is the contribution beyond InterMoE?

While InterMoE introduces dynamic expert routing, our method further incorporates cooperative interaction-conditioned expert selection, jointly considering individual motion, textual semantics, and partner motion. Unlike routing based on individual motion and text alone, our Cooperative Router explicitly incorporates partner-dependent information into expert assignments.

Moreover, STC-VAE dynamically integrates each local skeletal region with global body information, allowing different body parts to adaptively incorporate global context and learn richer, more expressive latent motion representations. This provides a stronger foundation for the interaction modeling performed by Inter-DiT, making the two components complementary.


Question 3: Is the improvement attributable to global–local fusion or increased model capacity?

Thank you for this important question! Our spatial fusion dynamically integrates global body posture and local joint dynamics at each timestep, allowing each local skeletal region to incorporate global contextual information while preserving fine-grained motion details.

To evaluate its effectiveness, we conducted ablation studies comparing global-only, local-only, and dynamic global–local fusion strategies. As shown in Table 4, the complete fusion achieves an FID of 4.658, outperforming both global-only (5.898) and local-only (4.814) representations. These results demonstrate the benefit of combining global and local spatial information.

MethodTop-1 ↑Top-2 ↑Top-3 ↑FID ↓MM-Dist ↓
Global Spatial0.5010.6540.7305.8983.766
Local Spatial0.4940.6480.7264.8143.772
Global–Local Fusion0.5150.6690.7444.6583.765
AuthorsOpenReview ↗

Author Rebuttal (2/3)

Question 4: How significant are the performance improvements over InterMoE?

Thank you for this important question! In quantitative evaluations, HHI-Dyn achieves better FID and R-Precision Top-1 on InterHuman, demonstrating competitive generation quality. Although our FID is slightly worse on InterX, HHI-Dyn outperforms InterMoE in R-Precision Top-1, Top-2, Top-3, and MM-Dist, indicating stronger text–motion semantic alignment.

Furthermore, our qualitative comparisons and user study provide additional evidence of improved motion realism and text–motion consistency, capturing perceptual differences that may not be fully reflected by FID alone. Detailed visualizations and user study results are provided in the supplementary material.

MethodInteraction Quality ↑Adherence to Text ↑
InterMoE3.633.56
HHI-Dyn4.064.30

Overall, HHI-Dyn demonstrates advantages in semantic alignment and perceptual motion quality, supporting the effectiveness of our proposed framework.


Question 5: Why use cosine similarity and causal encoding?

Thank you for this important question! Different interaction patterns involve varying degrees of mutual influence between individuals. In closely coordinated interactions, individuals often exhibit stronger spatial and motion correlations, which may be reflected in their learned feature representations. Therefore, cosine similarity provides a useful feature-level correlation signal rather than a direct measure of interaction strength. Specifically, the Partner Router learns partner-dependent features, while cosine similarity adaptively modulates their contribution to expert selection. Our ablation study further validates its effectiveness, as removing cosine similarity increases FID from 4.658 to 4.886.

Regarding causal encoding, we introduce it as a temporal inductive bias to preserve the sequential structure of motion representations. By encoding each frame based on preceding temporal information, causal encoding encourages temporally consistent feature learning and reduces reliance on future frames. Although bidirectional encoding is also suitable for offline motion generation, our causal design provides a structured way to capture temporal dependencies.

Ablation study of causal encoding:

MethodTop-1 ↑Top-2 ↑Top-3 ↑FID ↓MM-Dist ↓
w/o Causal Encoding0.4930.6340.7165.3613.774
Ours Full0.5150.6690.7444.6583.765

Ablation study of cosine similarity:

MethodR-Precision Top-1 ↑FID ↓MM-Dist ↓
w/o Cosine Similarity0.5064.8863.768
Ours Full0.5154.6583.765

Question 6: How does HHI-Dyn differ from ARMFlow in terms of causal modeling and the generation paradigm?

Thank you for this clarification! Although both methods employ causal convolutions, they address fundamentally different tasks and adopt distinct generation frameworks.

Specifically, HHI-Dyn focuses on text-driven human-human interaction generation, where both individuals' motion sequences are jointly generated from a textual description. We introduce causal convolutions in the first-stage STC-VAE to enhance temporal motion representations and improve subsequent diffusion-based generation. In contrast, ARMFlow focuses on reaction generation, where one individual's motion is generated conditioned on the other individual's motion and textual guidance.

Furthermore, HHI-Dyn employs 50-step DDIM sampling, rather than MeanFlow or one-step generation. Therefore, the similarity between HHI-Dyn and ARMFlow is primarily limited to the use of causal temporal modeling, while their task formulations, model architectures, and generation paradigms differ substantially. We have included ARMFlow in the related work.

AuthorsOpenReview ↗

Author Rebuttal (3/3)

Question 7: Why does HHI-Dyn not include physical plausibility metrics, such as penetration, contact consistency, or inter-person collision measures?

Thank you for this valuable suggestion! Our work primarily focuses on kinematics-based human-human interaction generation, following the evaluation protocol established by InterGen [1], which emphasizes motion quality, diversity, and text–motion alignment. Accordingly, we adopt the widely used kinematic evaluation metrics to ensure fair comparisons with existing methods.

We acknowledge that physical plausibility is an important aspect of human-human interaction generation that cannot be fully assessed by these metrics. In future work, we plan to incorporate physics-based interaction modeling, inspired by approaches such as InterAgent [2], to improve physical consistency and introduce quantitative evaluations of penetration, contact consistency, and inter-person collisions.

References

[1] Liang, H., et al. InterGen: Diffusion-Based Multi-Human Motion Generation Under Complex Interactions. IJCV, 2024.

[2] Li, B., et al. InterAgent: Physics-Based Multi-Agent Command Execution via Diffusion on Interaction Graphs. CVPR, 2026.


Question 8: What does "reduced implicit bias" mean in the qualitative comparison, and how is it evaluated?

Thank you for pointing out this ambiguity! In our qualitative comparison, "reduced implicit bias" refers to fewer visually observable deviations from plausible and semantically consistent interaction motions, rather than a formally defined or quantitatively measured bias.

Specifically, we use this term to describe qualitative differences in motion naturalness, body posture, and interaction consistency, following the qualitative discussion in Text2Interact [1]. We acknowledge that this expression may be misleading without an explicit definition or metric. Therefore, we will replace "reduced implicit bias" with "improved visual plausibility" to more accurately reflect our qualitative observations without implying a quantitatively validated improvement.

Reference

[1] Wu, Q., et al. Text2Interact: High-Fidelity and Diverse Text-to-Two-Person Interaction Generation. ICLR, 2026.


Question 9: Could the authors provide more details about the model architecture and training configuration?

Thank you for your valuable suggestion! We have provided additional details on the model architecture, implementation, and training configuration in Appendix C.2 (Training Details) to improve the clarity and reproducibility of our work. Please refer to this section for further information.

Reviewer_VYmpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HHI-Dyn is a latent diffusion model for text-conditioned two-person motion generation. Its VAE combines global per-frame motion features with local skeletal features through a learned spatial gate. The diffusion transformer augments self- and cross-attention with a mixture of experts whose routing depends on the motion, text, and partner features, with cosine similarity modulating the partner contribution. The paper evaluates on InterHuman and InterX, provides component ablations, and reports a user study on interaction quality and text adherence.

Critical strengths
  1. Combining local skeletal detail with global posture is a reasonable way to improve motion representations. Explicit partner conditioning also provides a useful design direction for interaction generation.

  2. The paper includes two datasets, component ablations, a user study, and implementation code. These provide useful evidence and make the proposed design easier to inspect.

Critical weaknesses
  1. The motivation for the cooperative router is not fully convincing. Cross-attention is already input-dependent, yet the paper does not identify a specific limitation that prevents it from adapting to partner influence. Moreover, latent-feature cosine similarity is not inherently a measure of interaction dependence: complementary actions such as attack and defense can be tightly coupled.

  2. The contribution beyond InterMoE appears incremental. InterMoE already uses text/motion-conditioned dynamic expert routing; HHI-Dyn mainly adds a partner-conditioned routing term and gated global–local fusion. The VAE fusion operates within each person, leaving its specific contribution to modeling inter-person dependence insufficiently explained.

Questions for authors
  1. What specific limitation of existing cross-attention prevents adaptation to varying partner influence, and how does the cooperative router address it?

  2. Why should cosine similarity in the learned latent space reflect interaction dependence, particularly for complementary actions? Is this interpretation encouraged by the learning objective or assumed by the router design?

  3. Beyond adding partner features to InterMoE-style routing, what new interaction-modeling principle is introduced? How does the per-person global–local fusion specifically address inter-person dependence?

5 comments
AuthorsOpenReview ↗

Author Rebuttal

Thank you for your thoughtful review. We appreciate your recognition of our global–local spatial fusion, partner-aware interaction modeling, and comprehensive experiments. We address your concerns regarding the cooperative router, cosine similarity, and methodological novelty below.

Question 1: What specific limitation of existing cross-attention prevents adaptation to varying partner influence, and how does the cooperative router address it?

Thank you for your insightful question! We agree that cross-attention is input-dependent and can adaptively exchange information between individuals. However, cross-attention primarily performs feature aggregation, whereas our Cooperative Router explicitly incorporates partner information into expert selection.

Specifically, our router jointly leverages individual motion, textual semantics, and partner motion to dynamically select specialized motion-pattern experts. These two mechanisms are complementary rather than interchangeable. Furthermore, our ablation studies show that removing the Partner Router leads to performance degradation, demonstrating the effectiveness of partner-aware routing beyond cross-attention alone.

MethodR-Precision Top-1 ↑FID ↓MM-Dist ↓
w/o Partner Router0.4975.1463.771
Ours Full0.5154.6583.765

Question 2: Why should cosine similarity in the learned latent space reflect interaction dependence, particularly for complementary actions?

Thank you for raising this important point! Different interaction patterns involve varying degrees of mutual influence between individuals. In closely coordinated interactions, the two individuals often exhibit stronger spatial and motion correlations, which may be reflected in their learned feature representations. Therefore, cosine similarity provides a useful signal for capturing feature-level correlations associated with interaction dynamics.

Importantly, we use cosine similarity not merely to estimate interaction strength, but also to adaptively modulate partner-aware expert routing. The Partner Router learns partner-related motion features through the generation objective, while cosine similarity adjusts their contribution to expert selection. This allows the routing mechanism to incorporate both learned partner information and feature-level correlations, including in complementary interactions where high similarity is not guaranteed. Furthermore, our ablation study shows that removing cosine similarity leads to performance degradation, supporting its empirical effectiveness.

MethodR-Precision Top-1 ↑FID ↓MM-Dist ↓
w/o Cosine Similarity0.5064.8863.768
Ours Full0.5154.6583.765

Question 3: Beyond adding partner features to InterMoE-style routing, what new interaction-modeling principle is introduced? How does the per-person global–local fusion specifically address inter-person dependence?

Thank you for this insightful question! Our main contribution lies in combining global–local spatial representations with cooperative interaction-aware expert routing, rather than simply incorporating additional partner features.

Specifically, STC-VAE dynamically fuses global body structure and local joint dynamics to learn more expressive motion representations for each individual, providing a stronger foundation for interaction modeling. For example, in tug-of-war, local joint movements such as gripping and leg bracing must be coordinated with global body posture and the spatial relationship between individuals. Modeling only local features may overlook the overall backward lean and coordinated pulling dynamics, highlighting the importance of global–local fusion.

MethodTop-1 ↑Top-2 ↑Top-3 ↑FID ↓MM-Dist ↓
Global Spatial0.5010.6540.7305.8983.766
Local Spatial0.4940.6480.7264.8143.772
Global–Local Fusion0.5150.6690.7444.6583.765

Building upon these representations, Inter-DiT jointly leverages individual motion, textual semantics, and partner motion for expert selection, with cosine similarity adaptively modulating the partner contribution to capture feature-level correlations between individuals.

Importantly, our ablation studies demonstrate that incorporating partner features improves generation performance, validating the effectiveness of cooperative routing. Furthermore, dynamic global–local fusion outperforms global-only and local-only representations in both reconstruction and generation, demonstrating its effectiveness in learning motion representations suitable for human-human interactions.

Thus, STC-VAE enhances individual motion representations, while Inter-DiT explicitly models inter-person dependencies, jointly improving interaction generation quality.

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely appreciate the time you took to review our manuscript and provide insightful comments. We hope our point-by-point response to your feedback addresses your concerns. We are writing to respectfully inquire whether our responses have successfully resolved all your questions and concerns. Please let us know if there is any additional information you require from our side.

Thank you very much!

Dear reviewer, the authors have responded to your comment. Could you please take a look? Let us know if you have further concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

[Summary to AC] Key Strengths and Rebuttal Clarifications

Dear Area Chair of Submission 19511,

We sincerely thank you for overseeing the review process. As the discussion period concludes, we would like to briefly summarize the key contributions and experimental evidence supporting HHI-Dyn.

1. Methodological Novelty and Motivation.

HHI-Dyn addresses text-driven human-human interaction generation through two complementary designs. To the best of our knowledge, we are the first to introduce dynamic global–local spatial fusion in this task. Specifically, STC-VAE adaptively integrates global body context into local skeletal representations, rather than modeling them independently.

Furthermore, Inter-DiT introduces cooperative partner-aware expert routing, jointly conditioned on individual motion, partner motion, and textual semantics, with cosine similarity modulating partner contributions. These designs are motivated by the need for expressive motion representations and interaction-aware generation, rather than simply combining existing components.

Our ablation studies support their effectiveness:

ConfigurationFID ↓
Global Spatial Only5.898
Local Spatial Only4.814
Without Partner Router5.146
Without Cosine Similarity4.886
HHI-Dyn (Full)4.658

2. Competitive Performance and Text–Motion Alignment.

HHI-Dyn achieves the best reported FID on InterHuman (4.658), outperforming InterMoE (4.677), along with better R-Precision Top-1 (0.515 vs. 0.512).

On InterX, despite a slightly higher FID (0.314 vs. 0.297), HHI-Dyn outperforms InterMoE across all R-Precision Top-1/2/3 and MM-Dist metrics, demonstrating better text–motion alignment.

MethodTop-1 ↑Top-2 ↑Top-3 ↑MM-Dist ↓
InterMoE0.4270.6120.7213.605
HHI-Dyn0.4280.6190.7293.602

Our user study further supports the improvements:

MethodInteraction Quality ↑Adherence to Text ↑
InterMask3.233.10
HiTMM3.533.46
InterMoE3.633.56
HHI-Dyn4.064.30

3. Qualitative Improvements and Reproducibility.

We particularly encourage the Area Chair to examine the comparative visualization demos in our supplementary material. These comparisons illustrate more natural two-person interactions, better motion coordination, and stronger adherence to textual descriptions. Such perceptual improvements cannot be fully captured by numerical metrics alone.

Importantly, we also provide implementation code alongside the visualization demos, supporting reproducibility and allowing further examination of our proposed framework.

4. Rebuttal Clarifications and Limitations.

We have carefully addressed the reviewers' concerns regarding methodological novelty, distinctions from InterMoE and DHVAE, component effectiveness, and evaluation.

Overall, HHI-Dyn combines a task-specific dynamic global–local fusion mechanism with cooperative partner-aware routing, supported by competitive benchmark results, extensive ablations, user-study evidence, and qualitative demonstrations.

We sincerely appreciate the reviewers' constructive feedback and your consideration. We hope these clarifications and experimental results will support a favorable assessment of our work.

Best regards,

The Authors of Submission 19511

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.