acceptodds
Under review as a conference paper at ICLR 2027

Beyond MoCap: Scaling Motion Tokenizers with Synthetic Human Motion for Generative Modeling

Abstract

Human motion generation models are fundamentally constrained by the limited diversity of motion capture datasets, which predominantly contain common, repetitive actions and fail to cover the long tail of complex human movements, resulting in a restricted motion vocabulary in learned latent representations and poor generalization to rare, compositional, and highly dynamic motions. In this work, we propose a framework for expanding the motion representation space by leveraging large-scale synthetic human motion, introducing a data generation pipeline that produces diverse, physically plausible motion sequences beyond the distribution of existing datasets and integrating it with a redesigned VQ-VAE tokenizer that adapts to this expanded motion space. Unlike conventional tokenizers trained on narrow data distributions, our approach jointly scales both the training distribution and the discrete codebook, enabling the model to capture a significantly richer set of motion primitives. We demonstrate that training with synthetic motion substantially improves the coverage and compositionality of the learned motion vocabulary, leading to consistent gains across motion generation tasks such as text-to-motion and motion continuation, while remaining fully compatible with existing frameworks including MotionGPT. Our results suggest that the primary bottleneck lies in the limited support of the learned motion representation, rather than model architecture alone. Scaling synthetic motion in tandem with representation learning offers a principled path toward more expressive, controllable, and generalizable human motion synthesis.

Primary area:
generative models
Keywords:
human motion generation, human motion representation, generative modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KsJzOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that the motion tokenizer rather than the generator is the key bottleneck in discrete-token human motion generation. To prove this claim, the authors propose to jointly scale the training motion distribution and the tokenizer capacity: they introduce a synthetic motion pipeline that generates new poses via crossover and mutation operations on a hierarchical bone representation, filters them with a pose prior, and builds sequences by interpolation, recovering root trajectories with a lightweight Transformer. The augmented tokenizer is trained on a mixture of real HumanML3D and synthetic motions with an enlarged codebook, and existing autoregressive generators are re-tokenized and fine-tuned. Experiments report reduced reconstruction FID and consistent gains across downstream backbones.

Critical strengths

[Originality]

  • Reframing the problem around the tokenizer as the bottleneck is a fresh perspective. I like this way of thinking about the problem.
Critical weaknesses

n/a

Questions for authors

[Ablation]

  • The ablation shows an interesting pattern: scaling the codebook alone or the synthetic data alone each hurts the quality, while the two together help. I'm curious if the authors have thoughts on why this contradictory pattern happens.

Other than this minor question, I don't have other concerns regarding this submission. It's an interesting reading and I enjoy how the authors formulate the problem. I thus lean towards accept.

1 comment

Overall

We thank all reviewers for their careful and constructive feedback. We are encouraged that Reviewers v9Fy and KsJz found our tokenizer-bottleneck framing well motivated, and that Reviewers scq5 and KsJz recognized the value of our reusable approach. We are particularly grateful to Reviewer KsJz for the encouraging assessment of our method.

To address the central concern shared by Reviewers scq5 and v9Fy—the decline in text alignment accompanying FID improvements—we introduce a simple supplementary RL-controlled motion correction stage. Our T2M-GPT experiments show improved text-alignment metrics while keeping FID changes within the predefined tolerance (see our response to Reviewer v9Fy, W1). We also address the remaining comments point by point below. We will incorporate these new results and discussions into the final version of the paper.

Q1 Ablation analysis of data scaling and codebook expansion

Thank you for drawing attention to this observation. We believe it reflects how well the training motion distribution matches the tokenizer's representational capacity. When only the codebook is enlarged, the ablation study in T2M-GPT also reports a similar non-monotonic trend: among 256, 512, and 1024 codes, 512 yields the best reconstruction FID. This indicates that increasing the number of codes does not necessarily improve performance under the original data setting. Our interpretation is that, with a limited motion distribution, the additional capacity may be difficult to fully utilize.

With synthetic motion data expansion alone, the richer motion patterns may exceed the effective capacity of a fixed 512-code representation. Joint scaling increases both motion coverage and representational capacity, potentially alleviating these two forms of mismatch. The controlled comparisons in Table 2 of our paper support the complementarity of synthetic motion data expansion and codebook capacity expansion under the current experimental setting.

Reviewer_v9FyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Current motion AI struggles with rare, complex moves because real mocap data is dominated by common repetitive motions. To fix this, the authors generate synthetic, physically plausible motions using an evolutionary pipeline to expand the model's motion vocabulary. Integrating this richer tokenizer into existing models visibly improves realism and handles tricky long-tail actions without changing the generator architecture.

Critical strengths
  1. Instead of just increasing model size, the paper solves the vocabulary limit caused by repetitive MoCap data.
  2. The upgraded tokenizer works with existing models like T2M-GPT and MotionGPT without any architectural changes.
  3. Detailed joint analyses confirm the pipeline reaches unvisited pose spaces while maintaining physical plausibility.
Critical weaknesses
  1. While physical realism improves, standard text-matching metrics (R-Precision and Multimodal Distance) degrade across all three backbones on HumanML3D.
  2. The paper explains away the benchmark alignment drop by arguing the standard HumanML3D evaluator is biased against expanded motion distributions. Yet, it relies on that exact same evaluator to claim superior alignment on 100 out-of-distribution prompts, creating circular reasoning.
  3. Generating sequences by simply interpolating (SLERP) between two static poses misses realistic human momentum, energy transfer, and ground contact.
Questions for authors
  1. You attribute the drop in benchmark text alignment (Table 3) to distribution bias in the HumanML3D evaluator. Why, then, is this exact same evaluator trusted to prove superior alignment on 100 out-of-distribution prompts in Table 4?
  2. Quadrupled the codebook to 2048, but the synthetic data contains no text descriptions. Without text pairs for these new primitives, isn't the drop in R-Precision simply the generator struggling to map text to an ungrounded token space?
  3. Rotational interpolation between static poses causes an 88% collapse (from 6.477 to 0.804) in acceleration dynamics and increases foot sliding (Table 6). Does this risk encoding unnaturally floaty transitions, and do you have perceptual user study results to confirm that humans actually prefer these motions over baselines?
2 comments

Overall

We thank all reviewers for their careful and constructive feedback. We are encouraged that Reviewers v9Fy and KsJz found our tokenizer-bottleneck framing well motivated, and that Reviewers scq5 and KsJz recognized the value of our reusable approach. We are particularly grateful to Reviewer KsJz for the encouraging assessment of our method.

To address the central concern shared by Reviewers scq5 and v9Fy—the decline in text alignment accompanying FID improvements—we introduce a simple supplementary RL-controlled motion correction stage. Our T2M-GPT experiments show improved text-alignment metrics while keeping FID changes within the predefined tolerance (see our response to Reviewer v9Fy, W1). We also address the remaining comments point by point below. We will incorporate these new results and discussions into the final version of the paper.

W1: Degradation of R-Precision and MM-Dist issue

We acknowledge that our current method improves FID while reducing R-Precision and increasing MM-Dist. Based on the experiments and analyses in our paper, we believe that this degradation stems partly from the absence of text descriptions for the synthetic motions and largely from evaluator bias. To directly address this trade-off, we add a simple RL-controlled motion correction stage to our existing method and have completed experiments with T2M-GPT.

Improving text alignment. We train a text–motion alignment judge on the HumanML3D training split. Starting from a single generated motion for each prompt, we use gradients from this judge to refine the motion in continuous latent space, bringing its features closer to those of the corresponding text. A small RL controller trained with proximal policy optimization (PPO) selects the correction step size, the strength of an anchor to the original motion feature, and when to stop. In other words, the judge provides the correction direction, while RL learns to control the correction process. The training reward encourages both better text–motion retrieval and a lower matching distance.

Limiting FID degradation. We first bound the magnitude of latent changes and anchor the corrected motion features to the original generated features to limit distribution drift. These local constraints alone do not guarantee FID preservation, so we also apply an actual distribution check: after each policy update, we recompute FID over the complete motion cohort drawn from the training split for each sampling seed. An update is accepted only if the FID change stays within the predefined budget for every seed; otherwise, it is rejected and rolled back.

Relative to the synthetic-data tokenizer model, the added correction stage preserves strong FID performance while improving Top-1 R-Precision and reducing MM-Dist (Table 1).

MethodFID ↓Top-1 ↑Top-2 ↑Top-3 ↑MM-Dist ↓DiversityMModality ↑
T2M-GPT0.1160.4910.6800.7753.1189.7611.856
T2M-GPT (synthetic data)0.0970.4640.6470.7473.2719.5942.266
T2M-GPT (synthetic data + RL)0.1030.5040.7020.7982.9589.3341.78

Table 1. T2M-GPT with synthetic-data scaling and RL-controlled correction. The first two rows reproduce the means from Table 3 of the paper. The correction judge is trained on the HumanML3D training split.

The added RL stage builds on the strong FID performance already achieved by our synthetic-data tokenizer. We will append this stage to the method section and update the result tables to distinguish the contributions of synthetic-data expansion and RL correction.

W2&Q1: Evaluator bias and the interpretation of OOD results

We acknowledge that the HumanML3D evaluator may exhibit distribution bias, and we do not claim that such bias is entirely eliminated in Table 4. However, unlike the standard HumanML3D benchmark in Table 3, Table 4 evaluates both methods on the same 100 OOD prompts that fall outside the original HumanML3D text distribution. We believe this setting provides a relatively fair comparison of their generalization capabilities, as both methods are evaluated under the same distribution shift using an identical evaluator and evaluation protocol. Therefore, we consider these results useful complementary evidence of relative performance on OOD prompts, while recognizing the limitations of the evaluator. We will explore more suitable benchmarks for evaluating OOD text-motion alignment in future work.

W3&Q3: Physical plausibility concern of the synthesized motions.

Real human motion involves complex dynamics, including acceleration, deceleration, momentum, and physical contact. Since SLERP directly interpolates joint rotations to produce smooth transitions, the substantial reduction in acceleration is expected. We also acknowledge the increased foot sliding as a limitation of our interpolation-based approach.

Our tokenizer uses fourfold temporal downsampling, corresponding to a token temporal resolution of four frames; encoding and decoding still aggregate neighboring temporal context. Local poses and short-term dynamics are important components of its representation learning. For unnaturalness that primarily concerns the long-range organization of complete motions, we expect a more limited impact on the local motion representations learned by the tokenizer. This expectation requires further quantitative validation. Beyond the existing metrics in Table 6, we have not conducted further quantitative plausibility analyses.

Although we have not conducted a formal perceptual user study, our qualitative video comparisons provide visual evidence of improved motion quality over the baselines. In our inspection of these examples, we did not observe obvious unnaturally floaty transitions. We acknowledge that these observations do not replace a systematic human evaluation, which we consider an important direction for future work.

Q2: Text grounding of the expanded codebook

We agree that expanding the motion vocabulary requires the generator to learn updated correspondences between text and motion tokens, which may affect text alignment.

However, the absence of text descriptions for synthetic motions does not mean that new codes cannot receive paired supervision. During downstream training, the motions in real motion–text pairs are re-encoded using the expanded tokenizer. New codes can therefore receive contextual supervision through real motions and their associated captions.

The token usage analysis in Appendix I shows that 1756 of the 2048 codes in the expanded codebook are activated by text-paired training motions, and 99.83% of generated tokens belong to this set. This indicates that the generator rarely uses codes that have never appeared in the paired training data.

To address the observed alignment degradation, we have further introduced a supplementary RL correction method; please refer to our response to W1.

Reviewer_scq5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The proposed method augments HumanML3D with synthetic sequences produced through pose recombination/mutation, anatomical filtering, spherical interpolation, and learned root recovery. Real and synthetic motions are used to train a larger VQ-VAE codebook, after which existing generators are fine-tuned on re-tokenized original text-motion pairs, without synthetic captions.

Critical strengths

The augmented tokenizer can be reused across several existing motion-generation frameworks without requiring text annotations for the synthetic motions.

Critical weaknesses
  • I feel like the method is not strongly motivated and carefully designed. The proposed synthetic data generation is essentially a motion data augmentation but with a specialized model to force certain "plausibility". Although this helps scale the data in amount, it could harm the text alignment performance as there's no such gaurantee that the synthesized data is semantically aligned with text or even reasonable. Previous works have already explored more systematic way to resolves issues like this, such as utilizing motion data estimated from real video data, or LLM-supervised motion data generation. Yet the paper failed to discuss about or compare with any of those.

  • The experiment results actually reflect the issue mentioned above. Better FID comes with worse text alignment on HumanML3D. For MotionGPT, Top-1 R-Precision falls from 0.492 to 0.432, while MM-Dist worsens from 3.096 to 3.578.

  • The baseline comparison in Table 3 is not convincing. The proposed method doesn't outperform MoMask on several metrices, yet the authors skip discussion about this. Most of the baselines are not latest works. It seems like a lots of competitive recent works are missing. And the paper does not explicitly justify selecting the three major backbones instead of the others.

  • The visual demo quality is not satisfying. There are noticeable artifacts such as jittering and abrupt pose changes in several samples, e.g. Tango step.

Questions for authors

See weaknesses

3 comments

Overall

We thank all reviewers for their careful and constructive feedback. We are encouraged that Reviewers v9Fy and KsJz found our tokenizer-bottleneck framing well motivated, and that Reviewers scq5 and KsJz recognized the value of our reusable approach. We are particularly grateful to Reviewer KsJz for the encouraging assessment of our method.

To address the central concern shared by Reviewers scq5 and v9Fy—the decline in text alignment accompanying FID improvements—we introduce a simple supplementary RL-controlled motion correction stage. Our T2M-GPT experiments show improved text-alignment metrics while keeping FID changes within the predefined tolerance (see our response to Reviewer v9Fy, W1). We also address the remaining comments point by point below. We will incorporate these new results and discussions into the final version of the paper.

W1: Motivation concern

Our core motivation is to synthesize richer motion primitives and jointly expand the tokenizer's motion coverage and codebook capacity, thereby improving downstream motion generation.

We agree that video-based motion extraction and LLM-supervised motion generation share our goal of improving motion generation models. Different data sources or supervision do not by themselves make one approach more systematic. Video-based extraction primarily recovers additional real motions for training, whereas our study directly targets the tokenizer's motion coverage and representation capacity. For LLM methods that rely on existing motion libraries or fixed motion representations, the motions they can realize remain constrained by the coverage of the underlying motion primitives. We aim to expand this coverage at the representation level.

We therefore view these methods as relevant and complementary research directions. We acknowledge that our related-work discussion is incomplete and will add these studies in the revision, clarifying their research objectives and connections to our approach.

Regarding the concern that expanding motion representations may affect text alignment, our token analysis (refer to Appendix I) shows that nearly all generated tokens have been observed in text-paired training data, so tokens without such coverage contribute only a small fraction of the output; this does not rule out semantic misalignment. Besides, to address the observed metric degradation, our supplementary RL correction method achieves, in our T2M-GPT experiments, improved text-alignment metrics while keeping FID changes within the predefined tolerance (see W2).

W2: Degradation of R-Precision and MM-Dist issue

Please refer to our response to Reviewer v9Fy, W1.

W3: Baseline comparisons and backbone selection

Our objective is to improve existing generators by expanding their motion representation. The central comparison is therefore each backbone with its original tokenizer versus the same backbone with our augmented tokenizer.

We chose T2M-GPT, MotionGPT, and MotionAgent because they allow us to examine this substitution across different generation frameworks while retaining a compatible T2M-GPT-style VQ-VAE representation. Our empirical conclusions about transferability are therefore limited to this family of models.

Although these backbone comparisons support our core conclusion, we agree that Table 3 should also place these improvements in the context of stronger recent methods. MoMask uses a hierarchical residual-quantization representation. Applying our scaling strategy to this architecture requires adapting its tokenizer and retraining the corresponding generator. We will identify this extension as future work.

W4: Jittering and abrupt pose changes in the visual demonstrations

We acknowledge the jittering and abrupt transitions noted by the reviewer, including the Tango example. Our goal is to improve generation relative to the corresponding backbone, while these artifacts remain a limitation of the generation pipeline as a whole. Broader tokenizer coverage alone does not ensure temporally coherent or physically plausible complete motions. The tokenizer, decoder, and downstream generator may all contribute to these issues; the current experiments have not isolated their respective contributions.

Our method improves performance over the original model, but does not fully eliminate jitter. We will explicitly acknowledge this limitation in the revised manuscript. Adapting the method to stronger architectures such as MoMask is an important research direction for addressing these remaining visual issues.

Reviewer_scq5OpenReview ↗

Response to the author

Thanks the authors for the response. However, I have to say that the response doesn't address my concerns much.

First of all, the motivation and overall method is still not convincing to me. There are quite a few positioning statements that are not well explained or supported by experiments just as in the paper. For example, " video-based extraction primarily recovers additional real motions for training" -- but isn't this intuitively much better than having synthetic data in the first place? And using VLM/LLM during learning or pre-learning annotation even can provide more semantically guaranteed real text-motion pairs that may comprehensively improve the FID and text-alignment at the same time. If not so, are there any experimental supports to prove that the proposed method can outperform them in certain aspects? Moreover, the qualitative results and the reasons for selecting the current backbones instead of the state-of-the-art ones is still not well explained or justified by additional experiments, which to me is also a key issue to the soundness of the paper.

For the above reasons, I will keep my score.

Thank you for further clarifying your concerns. We understand your view that synthetic data could potentially be replaced by other sources, such as video-derived or VLM-generated data. However, we would like to reiterate that the design of the synthetic data pipeline is only one part of our method. Strengthening the motion tokenizer and reusing the enhanced tokenizer across different generation frameworks are also central to our motivation.

Regarding the data, we applied the distributional analysis used in Fig. 3 of our paper to Motion-X+. The results show broader motion distributions in our synthetic data than in Motion-X+, suggesting that this diversity is useful for training an expanded motion vocabulary in the tokenizer. In addition, using Motion-X for downstream motion training still presents practical challenges. For example, GENMO[1] notes quality issues with Motion-X's 3D motions and therefore uses only its 2D keypoints for training.

BoneHumanML3D 50%Synthetic 50%Motion-X+ 50%HumanML3D 90%Synthetic 90%Motion-X++90%
Pelvis → Left Hip0.35%0.41%0.39%4.57%4.61%14.31%
Left Hip → Left Knee0.60%2.60%1.29%6.84%27.98%19.66%
Left Knee → Left Ankle0.53%2.10%0.87%6.19%27.65%11.37%
Left Ankle → Left Foot2.15%10.19%3.27%22.96%50.45%23.13%
Pelvis → Right Hip0.38%1.19%0.41%4.51%4.59%13.93%
Right Hip → Right Knee0.60%2.60%1.26%7.18%26.99%18.64%
Right Knee → Right Ankle0.56%2.18%0.88%6.71%27.60%12.06%
Right Ankle → Right Foot2.22%10.03%3.17%23.43%49.90%22.06%
Pelvis → Spine10.38%0.45%1.33%3.22%3.48%19.79%
Spine1 → Spine20.49%0.64%0.66%6.19%6.25%13.87%
Spine2 → Spine30.92%1.10%2.22%10.77%10.85%22.07%
Spine3 → Neck0.31%0.34%0.64%3.41%3.37%10.58%
Neck → Head1.95%2.24%2.18%21.38%21.15%20.04%
Spine3 → Left Collar0.51%1.02%0.57%6.98%13.85%7.06%
Left Collar → Left Shoulder1.99%3.21%3.06%25.61%40.12%24.69%
Left Shoulder → Left Elbow1.52%3.94%3.63%16.49%31.78%26.99%
Left Elbow → Left Wrist4.67%15.31%13.14%44.30%55.17%47.74%
Spine3 → Right Collar0.54%0.99%0.57%7.22%13.70%7.06%
Right Collar → Right Shoulder2.21%3.25%3.22%27.61%40.35%26.96%
Right Shoulder → Right Elbow1.86%4.74%3.70%18.48%33.56%28.04%
Right Elbow → Right Wrist5.28%13.49%16.35%45.35%55.45%48.69%

We acknowledge that additional text–motion pairs can enhance semantic supervision and complement our work on expanding motion representations. However, the current synthetic motion representation and available VLMs do not directly support automatic text–motion pairing in our setup. Such pairing is not required for our tokenizer enhancement, as the tokenizer is trained on motion data alone. Incorporating additional text–motion supervision into our framework remains an important direction for future work.

Backbone selection. We appreciate the reviewer's perspective. We believe our controlled comparisons across three backbones support the core contribution, while acknowledging that broader validation would strengthen the evidence. We respect this difference in assessment. Given the limited time available during the rebuttal period, we commit to including additional controlled experiments on a state-of-the-art backbone in the final version.

[1] Jiefeng Li et al. GENMO: A GENeralist Model for Human MOtion. ICCV, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.