acceptodds
Under review as a conference paper at ICLR 2027

Think with Motion: A Unified Understanding and Generation Framework for Individual and Interactive Human Motion

Abstract

Text-driven human motion modeling bridges high-level semantics with complex physical dynamics. For interactive motion generation, prevailing methods leverage coupled representations in the raw motion space, struggling to interpret underlying social intent through interactive text-conditioned descriptions. Meanwhile, emerging motion–language models unify motion understanding and generation, whereas the shared parameters across both tasks induce severe interference. To address these issues, we propose , a simple and unified architecture for individual and interactive motion understanding and generation. Specifically, we introduce a relation-decoupled representation that factorizes interactions into individual movements and relative geometry. further introduces a task-specific Mixture-of-Transformers architecture to mitigate interference between motion understanding and generation. We additionally establish a participant-level Chain-of-Thought reasoning mechanism, enabling motion understanding to explicitly guide interactive motion generation. Extensive experiments on HumanML3D and InterHuman datasets demonstrate that achieves superior performance across all motion modeling tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Motion Modeling, Human Motion Generation, Human Motion Understanding, Computer Vision, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Kqu7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes I²MoT, a unified framework for individual and two-person interactive motion generation and understanding. It contributes three components: CMR, A task-specific Mixture-of-Transformers, Participant-grounded chain-of-thought (CoT) supervision

The paper claims that this design mitigates task interference, lets understanding guide interactive generation, and achieves superior performance across all tasks on HumanML3D and InterHuman.

Critical strengths
  1. The participant-level annotation of InterHuman (6,217 pairs, multi-view with human review) is a potentially useful resource for fine-grained interaction modeling.
  2. The paper proposes an evaluation method on unified model: freezing the language branch to preserve LLM’s semantics (including CoT). We find this valuable and reasonable. The paper is transparent about evaluation settings and includes informative diagnostics, such as the motion/relation substitution analysis in Table 13.
Critical weaknesses
  1. The central claim of this paper, that unified model have participant-level CoT (benefied from motion understanding) that guides interactive motion generation, is not supported by either the design or the experiments. CoT is introduced as a semantic foundation for understanding-guided generation and as supporting complex interaction generation through individual motion understanding, due to the unified design proposed by this paper. However, Appendix A.9 states: The generation task retains the original interaction caption as its condition; the structured output is not fed back as a generation plan. Therefore, it's not clear how CoT functions in this unified model. Moreover, we didn't see the ablation study w/o CoT in terms of motion understanding/generation.

  2. The claimed benefits of unification, mitigating task interference (Q2) and understanding-guided generation (Q3), are tested only on jointly trained models, which perform substantially worse than the models reported in the main tables. Table 1 (generation) uses a generator trained only for text-to-motion; and Table 2 (individual understanding) uses the Qwen3.5 VAE captioner. Neither table/figure reports performance on separate models. Performance of Q2 and Q3 on these models should also be reported (even if very low performance), to make sure the benefits are emerged from model unification.

Questions for authors

Can you provide a controlled comparison of interaction understanding w/o participant-grounded CoT, using the same data and evaluation set? In Table 9, the CoT variant has lower R@3 than non-CoT (0.627 vs. 0.659), while also differing in evaluation samples (857 vs. 1,082), training updates (26k vs. 342k), and relation projection (two-stage vs. direct). A matched ablation is needed to verify the benefit of the new annotations and the CoT formulation, and to justify them relative to existing participant-level descriptions such as in2IN.

Reviewer_567HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents I2I^2MoT for motion generation and understanding in both single-person and two-person interaction settings. It decomposes a two-person motion into two canonicalized individual motion sequences and their initial relative position and orientation, allowing both settings to share a motion variational autoencoder. With the pretrained language model and motion autoencoder frozen, the model uses generation and understanding experts with separate parameters: the generation branch jointly predicts motion latents and relative relations through flow matching, while the understanding branch produces textual descriptions from motion. The authors also add participant-level motion annotations to InterHuman and supervise understanding with a structure that first describes each participant's actions and then summarizes the overall interaction. Experiments on HumanML3D and InterHuman demonstrate a shared generator across single-person and two-person settings, single-person motion understanding, and joint training for two-person generation and understanding.

Critical strengths
  1. The unified representation is clearly structured and demonstrates model sharing across single-person and two-person settings.
  2. To address the lack of participant-specific descriptions in two-person data, the authors develop an annotation workflow combining observations of motion from multiple views, model-based review, and human verification.
Critical weaknesses
  1. Participant-level annotations and structured understanding supervision are important contributions of the paper, but the current experiments do not clearly establish their independent benefits. The authors disclose these differences and explain that structured supervision uses the subset with valid annotations. However, they do not provide a direct ablation on a shared dataset under matched settings, nor sufficiently explain why such substantially different comparison settings were adopted.
  2. The static images mainly use skinned rendering, whereas the MP4 and GIF files examined mainly show skeleton animations. The lack of continuous skinned animations corresponding to the static images makes it difficult to compare body surfaces, contact relationships, and details of continuous motion.
  3. Visible jitter can be observed in the supplementary animations examined, but the paper does not provide quantitative results specifically evaluating jitter or temporal smoothness.
Questions for authors
  1. Could the authors use the same training samples with valid participant annotations, hold the motion front end, relation mapping, training schedule, evaluation samples, and scoring implementation fixed, and compare only supervision using overall interaction descriptions against supervision using participant descriptions plus overall interaction descriptions, reporting both understanding and generation results?
  2. Please clarify the actual tasks and models associated with the identically numbered videos in Timeline and VAE_Recon. For the visible jitter, are there corresponding comparisons and quantitative analyses of ground-truth motions, VAE reconstructions, and full generation results that establish its severity and source?
  3. I recommend reorganizing the supplementary materials and adding continuous skinned animations.
Reviewer_cLZiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes I²MoT, a unified framework for individual and interactive human motion understanding and generation. The method introduces a Compositional Motion–Relation Representation (CMR) to decouple individual motions from inter-person spatial relationships, a task-specific Mixture-of-Transformers architecture to separate motion understanding and generation, and participant-grounded Chain-of-Thought (CoT) supervision for interaction understanding. The paper addresses a meaningful research problem and presents a reasonably comprehensive framework. However, I have concerns regarding the novelty of the proposed representation, the insufficient validation of the key architectural contributions, and the experimental evidence supporting the claimed benefits of unified understanding and generation.

Critical strengths
  1. Meaningful research problem. Unifying individual and interactive human motion understanding and generation is an important and challenging direction. The proposed framework provides a systematic approach to integrating these tasks.

  2. Reasonable architecture and representation. Separating individual motion representations from relative spatial relationships is intuitive and facilitates the reuse of single-person motion priors. The task-specific Transformer architecture is also a reasonable design for handling different modeling objectives.

  3. Relatively comprehensive implementation and documentation. The paper provides participant-level annotations, multiple quantitative comparisons, ablation studies, and detailed implementation information in the appendix, which are valuable for understanding the framework.

Critical weaknesses
  1. Insufficient methodological novelty and differentiation from prior work. The proposed CMR representation closely resembles the XH3D representation introduced in SocialGen, which already decomposes multi-human interactions into individual motion representations and initial relative spatial relationships. Furthermore, the task-specific Transformer architecture builds on existing motion-language frameworks such as MotionGPT3 and LLaMo. Although these works are cited, the paper does not sufficiently clarify the fundamental technical differences or demonstrate the advantages of its proposed representation over existing alternatives.

  2. The key claims regarding task decoupling and CoT-guided generation are not adequately validated. The authors argue that separating understanding and generation experts mitigates task interference, yet the ablation studies do not include a controlled comparison with a shared motion expert. More importantly, the participant-grounded CoT is used as supervision for motion understanding rather than being explicitly incorporated into the generation process. While indirect benefits through shared representations are possible, the paper does not provide a controlled comparison between joint training with and without CoT supervision. Consequently, it remains unclear whether the proposed reasoning mechanism actually improves motion generation.

  3. Insufficient experimental evidence for the unified framework and unclear qualitative presentation. The strongest generation results are obtained from a generation-only model, whereas the jointly trained understanding-and-generation model produces substantially different results under different training configurations. Although joint training slightly improves FID in Table 3, it reduces text-motion alignment, raising questions about the claimed mitigation of task interference. In addition, the evaluation lacks quantitative measurements of physical contact and interaction consistency. The supplementary qualitative results are also difficult to follow: Figure 9 contains inconsistent row descriptions, while Figure 10 has conflicting baseline labels between the figure and caption. The correspondence between supplementary videos, examples, and compared methods should be clarified.

Questions for authors
  1. What are the fundamental differences between CMR and the XH3D representation in SocialGen? Can the authors provide a controlled comparison to demonstrate the specific benefits of CMR?

  2. Can the authors provide controlled ablations comparing shared versus task-specific motion experts, and joint training with versus without participant-grounded CoT? Since the CoT descriptions are not explicitly used as generation conditions, what evidence demonstrates that they improve motion generation?

  3. Can the authors clarify which checkpoints support the different tasks and reported results, and whether a single checkpoint can achieve competitive performance across all four tasks? Additionally, can the authors clarify the supplementary method labels and video-to-example correspondence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.