acceptodds
Withdrawn from ICLR 2027

HuMoGPT: A Unified Foundation Model for Human Motion Understanding and Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human motion modeling remains fragmented across specialized tasks, including recognition, captioning, and generation. These tasks are typically addressed by separate models trained on heterogeneous datasets with incompatible representations and supervision, limiting the development of general-purpose motion models. We introduce HuMoGPT, a unified multimodal foundation model that brings diverse human motion understanding and synthesis tasks into a single framework. Our approach has three components. First, we develop MotionRxiv, a large-scale data-curation pipeline that harmonizes heterogeneous motion corpora into a common skeletal representation with consistent normalization, leakage-safe splits, and multitask annotations, yielding approximately M motion clips. We then use these clips to generate M skeleton-grounded question-answering pairs and K multi-turn dialogues. Second, rather than imposing a single representation for understanding and synthesis, we introduce two complementary motion interfaces: SeMoT, a semantic motion tokenizer contrastively aligned with a pretrained language model for understanding, and GeMoT, a continuous temporal autoencoder that preserves geometric fidelity for generation. Third, HuMoGPT integrates these interfaces into a pretrained multimodal language model, using semantic motion representations for understanding and flow matching in the continuous motion latent space for synthesis. Finally, modality-, task-, and domain-routed low-rank adapters enable parameter-efficient multitask learning. HuMoGPT supports captioning, question answering, dialogue, and recognition alongside conditioned generation, completion, in-betweening, and editing. Despite its broad task coverage and single 1.5B backbone, HuMoGPT achieves state-of-the-art motion–text retrieval on HumanML3D, the lowest text-to-motion FID among compared baselines, and captioning performance competitive with a specialized model five times its size.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Motion, Unified Model, Motion Understanding, Motion Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DaSHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents HuMoGPT, a unified model for human-motion understanding and generation. The system combines: (1) MotionRxiv, a curated corpus of approximately 1.8M motion clips with unified representations and additional synthesized supervision; (2) SeMoT, a semantic motion tokenizer aligned with language for understanding; (3) GeMoT, a continuous latent autoencoder for motion generation; and (4) a pretrained multimodal language-model backbone equipped with flow matching and modality/task/domain-routed LoRA adapters. The model supports captioning, retrieval, recognition, question answering, dialogue, text-to-motion generation, completion, in-betweening, editing, and interaction generation.

The paper reports strong results on HumanML3D retrieval, captioning, and text-to-motion generation, along with additional evaluations on recognition, grounded dialogue, completion, in-betweening, and editing. However, because HuMoGPT is trained on a substantially larger and more heterogeneous dataset than most compared methods, the current experiments do not clearly establish whether the reported gains arise from the proposed modeling choices or primarily from data scale and supervision. The paper itself acknowledges this confound for the SeMoT retrieval comparison.

Critical strengths

1. Large-scale motion-data aggregation and supervision.

The construction of MotionRxiv is potentially valuable. The authors aggregate heterogeneous motion sources into a common representation, construct leakage-aware splits, and introduce large-scale grounded QA/dialogue supervision. The appendix also provides useful discussion of representation conversion, normalization, filtering, and grounding.

2. Broad system capability.

HuMoGPT attempts to support an unusually broad range of motion understanding and synthesis tasks within one framework. The system achieves strong reported performance on several major benchmarks, particularly motion-text retrieval, captioning, and text-to-motion generation.

Critical weaknesses

1. Unclear architectural novelty and inadequate positioning against the closest prior work.

The architectural contribution appears limited relative to recent unified motion-language models. HuMoGPT combines a pretrained language backbone, continuous motion representations, separate understanding/generation pathways, continuous generative modeling, and staged multimodal training. These ideas substantially overlap with the design space of closely related unified models such as MotionGPT3, UniMotion, LLaMo, yet the manuscript does not clearly articulate the conceptual advance over such work. At the same time, the related-work section spends substantial space on systems such as MoChat and ChatPose that are substantially less comparable because they do not provide the same unified motion understanding-and-generation functionality.

This makes it difficult to identify the main technical contribution of the paper beyond system integration and scaling.

2. The experiments do not disentangle architectural improvements from massive increases in training data and supervision.

HuMoGPT and SeMoT are trained using a large heterogeneous corpus, whereas many baselines use substantially smaller benchmark-specific datasets. The authors explicitly acknowledge that the SeMoT comparison is not a controlled architectural comparison. Yet the same confound affects much of the paper. Crucially, the paper lacks controlled experiments using the same backbone, training data, and compute to establish the value of its major modeling choices. In particular, there is no convincing ablation showing that using separate SeMoT and GeMoT representations is superior to a shared motion representation. The routed-LoRA ablation is also reported primarily for text-to-motion rather than demonstrating consistent benefit across the claimed task taxonomy. This problem extends to the dataset contribution. Although the appendix explains parts of the conversion pipeline, the dataset is insufficiently characterized for something presented as a primary contribution: detailed source/task distributions, caption provenance and quality, annotation generation for sources without native text, source imbalance, duplicate statistics, and the effect of individual data sources on downstream performance are not systematically analyzed.

References:

  • Li et al., LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens, CVPR 2026;
  • Zhu et al., MotionGPT3: Human Motion as a Second Modality, ICLR 2026;
  • Wang et al., UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation, ECCV 2026.
Questions for authors

The work represents substantial engineering effort and data curation, but I do not find the manuscript sufficient for acceptance, at least in its current form. The closest prior unified motion-model literature is inadequately addressed, leaving the architectural novelty unclear. More importantly, the central empirical results are strongly confounded by differences in data scale and supervision, while the key proposed components are not validated through controlled experiments. Several advertised modalities and tasks also receive substantially less evaluation than the breadth of the claims suggests.

My questions for authors:

  1. Can the authors provide controlled experiments in which data, backbone, and training budget are fixed, while varying the proposed architectural choices? In particular, I would like to see:
  • a single shared motion representation vs. SeMoT + GeMoT;
  • shared adaptation vs. routed adaptation across multiple understanding and generation tasks;
  • training on benchmark-scale data vs. MotionRxiv-scale data.

Without such comparisons, I cannot determine whether the improvement is architectural or primarily due to data scaling. These are true interesting stories that worth dicussing.

  1. Since MotionRxiv is presented as a central contribution, can the authors provide a substantially more detailed characterization of its composition and supervision? In particular, how are text annotations obtained for every source, including video/motion datasets without native captions, what fraction of captions are human/generated/VLM-generated, and how are these sources weighted during training? Also, will the dataset be open-sourced? This could be a great contribution to the community.

  2. What measurable contribution does the video modality make? The architecture explicitly includes video-conditioned motion generation, but I could not find a corresponding controlled quantitative study and discussion showing whether video improves generation or other multimodal capabilities.

  3. Can the authors provide a stronger evaluation of motion editing that separately measures instruction satisfaction and preservation of attributes that should remain unchanged? In the supplementary examples, some edits appear achievable through simple temporal transformations, while timing edits are visibly more difficult. The paper itself reports a timing-edit example for which the prediction is worse than copying the source. How do the authors distinguish genuine semantic editing from simple speed/trajectory transformations?

  4. Regarding the suppementary video, is the system running in real-time? Or is the video processed and edited? For motion editing demonstration, it seems that only the frequency is faster but the root trajectory stays the same. This leads to serious drifting and unnatural motion.

Reviewer_s11EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HuMoGPT, a unified model for human motion understanding and synthesis. It introduces the MotionRxiv data pipeline, which consolidates approximately 1.8 million motion clips and generates about 2.9 million question-answering pairs and 45,000 multi-turn dialogues. HuMoGPT uses SeMoT to represent motion semantics and GeMoT to preserve continuous geometric information. Motion generation is performed through flow matching in the continuous latent space, while modality-, task-, and domain-routed LoRA adapters support retrieval, captioning, question answering, recognition, generation, completion, and editing.

Critical strengths
  1. Substantial data and system contribution. The scale of MotionRxiv and the joint training of diverse understanding and generation tasks provide considerable engineering and research value.
  2. Well-motivated dual-interface design. SeMoT and GeMoT separately target semantic understanding and geometric generation, avoiding the potential bottleneck of forcing both task families to share a single discrete representation.
Critical weaknesses
  1. Novelty and scale effects are not sufficiently disentangled. Prior work such as M³GPT has already unified motion understanding and generation. HuMoGPT’s main additions are larger-scale data, dual motion interfaces, flow matching, and routed LoRA adapters. However, the paper lacks direct comparisons under matched data, model size, and computational budgets, making it difficult to determine whether the gains arise from the proposed design or simply from scaling.
  2. Insufficient validation of data reliability and the “unified model” claim. The paper should provide clearer details on cross-dataset deduplication, subject-level separation, benchmark leakage checks, and quality control for automatically generated QA data. Moreover, if different tasks depend on different routed LoRA adapters, it remains unclear to what extent the resulting system constitutes a single shared model rather than a shared backbone with task-specific modules.
Questions for authors
  1. Under matched backbone, training data, and computational budgets, how much does HuMoGPT improve over M³GPT, a single-tokenizer design, and task-specific models?
  2. Do all tasks use the same checkpoint and adapters? How many task-specific parameters are introduced for each task?
  3. How does MotionRxiv perform cross-dataset deduplication, subject-level separation, and benchmark leakage detection? What human validation was conducted for the automatically generated QA pairs and dialogues?
  4. What are the individual contributions of SeMoT, GeMoT, flow matching, and routed LoRA?
Reviewer_BVVhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

HuMoGPT extends Show-o2, a unified motion–language model, with unified motion understanding and generation capabilities. The backbone is adapted with LoRA branches routed per token by modality and per sequence by task family and source domain, under a four-phase curriculum. Validation covers diverse motion generation and understanding tasks.

Critical strengths
  1. Careful, well-structured data curation and grounded language supervision. If released as promised, it has value independent of the specific model.
Critical weaknesses
  1. Incomplete evaluation and limited novelty.

    • Many capabilities listed in Table 3 are never evaluated, namely temporal action localization, action counting, action-to-motion, video-to-motion, and trajectory-conditioned synthesis, even though established baselines and evaluation protocols exist for them (e.g., BABEL for temporal localization, OmniControl for trajectory control). Two-person interaction is shown only qualitatively.
    • The choice of Show-o2 rather than a text-only pretrained LLM as the backbone is weakly motivated. Its visual pathway does not appear to be used, since video conditioning goes through a separate encoder (Sec. 6.3). No ablation shows that unified multimodal pretraining benefits motion tasks.
    • Modality-based routing is not a new idea [1, 2]. Specializing a frozen backbone through modality- or domain-specific LoRA adapters or experts is also a well-established approach [3, 4]. None of these works is cited or discussed, so the contribution of the hierarchical routed LoRA (Sec. 6.4) relative to prior art is unclear.
  2. Unfair comparisons. The main comparisons (Tables 2, 14 and 21) are against baselines trained only on HumanML3D, whereas HuMoGPT, GeMoT and SeMoT are trained on ≈1.8M clips. Without a same-data baseline (e.g., a baseline retrained on MotionRxiv, or HuMoGPT trained on HumanML3D only), the reported gains cannot be attributed to the proposed design rather than to the larger training corpus.


Reference: [1] Zhang, Juze, et al. "Vibes: A conversational agent with behaviorally-intelligent 3d virtual body." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [2] Li, Zekun, et al. "Llamo: Scaling pretrained language models for unified motion understanding and generation with continuous autoregressive tokens." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [3] Abouelenin, Abdelrahman, et al. "Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras." arXiv preprint arXiv:2503.01743 (2025). [4] Shi, Weijia, et al. "Flexolmo: Open language models for flexible data use." Advances in Neural Information Processing Systems 38 (2026): 165943-165974.

Questions for authors

Although HuMoGPT is presented as supporting a broad range of tasks, only a subset of them is evaluated. Moreover, the ablations are conducted on very few tasks rather than across the full taxonomy, which makes the claimed contributions unconvincing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.