acceptodds
Under review as a conference paper at ICLR 2027

Better Conditioning, Better Motion: Generation-Oriented Text Representation Learning for Motion Generation

Abstract

Text-to-motion generation aims to synthesize realistic human motion sequences from natural language descriptions. Existing methods typically use a text encoder separately pretrained on proxy objectives. However, such encoders struggle to capture subtle dynamic nuances, leading to inadequate text representations for motion generation. We argue that effective text representations for motion generation should be generation-oriented, i.e., optimized to better support the synthesis of high-fidelity, text-aligned motion sequences. Accordingly, we propose LeGO, a co-optimization framework that Learns Generation-Oriented text representations for motion generation and derives LeGO-CLIP, a motion-specialized text encoder. Specifically, LeGO adapts a pretrained CLIP text encoder into LeGO-CLIP by jointly optimizing it with a motion diffusion model, allowing the encoder to receive differentiable feedback from generation quality through two semantic consistency constraints: a motion consistency constraint and a text consistency constraint. Comprehensive experiments demonstrate the effectiveness of our method from three perspectives: 1) Performance: It achieves state-of-the-art results on HumanML3D, KIT-ML and SnapMoGen benchmarks; 2) Efficiency: It significantly accelerates convergence, requiring up to 8x fewer training iterations; and 3) Transferability: LeGO-CLIP serves as a plug-and-play replacement and consistently improves the performance of diverse motion generation models. The source code will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
text-to-motion generation, human motion generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LAaKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that contrastive pretraining optimizes text encoders for cross-modal alignment, a proxy objective that does not directly optimize their usefulness for motion synthesis. To address this objective mismatch, LeGO jointly trains a motion diffusion generator and adapts a pretrained text encoder through LoRA. Frozen text and motion projectors provide semantic consistency losses alongside motion reconstruction. The resulting LeGO-CLIP encoder can also be transferred to other motion generators. Experiments on HumanML3D, KIT-ML, and SnapMoGen evaluate generation performance, encoder transferability, and sensitivity to fine-grained instructions.

Critical strengths
  1. Well-motivated text encoder adaptation: The paper identifies a plausible mismatch between contrastive pretraining and downstream motion generation. Jointly adapting the text encoder with the motion generator is a reasonable approach to learning representations suited to the generation task.

  2. Comprehensive empirical evaluation: Experiments cover three datasets and demonstrate encoder transferability across different generation architectures. Physical controllability tests and a user study complement embedding-based metrics, providing broader evidence of improved motion quality and instruction following.

Critical weaknesses
  1. Insufficient positioning and comparison with related work: MATE [1] also improves text representations for fine-grained motion generation, but its relationship to LeGO is not discussed in the Introduction or Related Work. Although it appears in the benchmark comparisons, it is omitted from the semantic sensitivity analysis in Section 4.3. Clarifying the methodological differences and including this comparison would better establish LeGO’s contribution.

  2. Reliance on contrastively pretrained projectors: The paper argues that contrastive learning optimizes cross-modal alignment but may fail to capture subtle dynamic distinctions needed for motion generation. However, its consistency losses use contrastively pretrained text and motion projectors to guide joint training. These projectors may share the limitations that motivate the paper. The authors should explain why they provide suitable supervision for fine-grained dynamics and how joint training overcomes those limitations. Furthermore, although the paper addresses a meaningful problem, its methodological contribution seems somewhat incremental.

[1] Motion-Aligned Word Embeddings for Text-to-Motion Generation. ICLR, 2026.

Questions for authors
  1. Suitability of the supervision projectors: Why are the contrastively pretrained projectors suitable for supervising subtle dynamic distinctions, given the paper’s concern that contrastive learning may align text and motion at a coarse semantic level but overlook fine-grained motion details?

  2. Semantic sensitivity comparison: How does LeGO-CLIP compare with MATE [1] under the keyword-substitution protocol in Section 4.3?

  3. Loss formulation and performance claims: Minimizing Eq 2 as written would discourage matched-pair alignment. Is this a typesetting error? Also, does the state-of-the-art FID claim in Section 4.1 refer only to diffusion-based methods?

4 comments

[Q1]: Comparison with MATE on Semantic Sensitivity Analysis

We agree that including MATE in the semantic sensitivity analysis would provide a more comprehensive comparison. However, MATE has not released its official implementation or pretrained weights. Therefore, we reproduced the MATE text encoder strictly following the details provided in the paper and trained it on the HumanML3D training set. We have updated Figure 5 (3D Histogram) and Figure 6 (Text Length vs. Cosine Similarity) to include MATE, with the following observations:

  1. In Figure 5, MATE shows better separation across similarity ranges than CLIP and LAMP. However, it still produces more highly similar text pairs than LeGO-CLIP, indicating that LeGO-CLIP better distinguishes subtle semantic differences.
  2. In Figure 6, both MATE and LeGO-CLIP exhibit semantic dilution effect as expected: as text length increases, replacing a single word has less impact on the overall representation, leading to higher cosine similarity. However, LeGO-CLIP consistently maintains substantially lower similarity than MATE across nearly all text lengths, which demonstrate LeGO-CLIP's better discriminability across nearly all text lengths.

[Q2]: Suitability of Using Contrastively Pretrained Projectors

We thank the reviewer for this question.

First, our central argument is that the texet encoder trained with cross-modal alignment alone does not necessarily yield text representations optimal for downstream motion generation.

Although contrastively pretrained projectors may not capture every fine-grained dynamic distinction, their strong text-to-motion and motion-to-text retrieval performance (Appendix I, Table 10) demonstrates effective discrimination between matched and mismatched text-motion pairs, supporting the construction of a shared semantic space for consistency supervision.

To compensate for potential limitations in the projectors' fine-grained distinction, LeGO combines high-level consistency constraints with element-wise MSE reconstruction loss. Fine-grained dynamic differences, such as variations in walking speed, may not be fully captured in the projector space but can still be penalized by MSE loss in the original motion space. Thus, rather than requiring the projectors to capture every fine-grained distinction, joint training enables LeGO-CLIP to learn generation-oriented representations from complementary semantic and reconstruction supervision.

To examine the contribution of reconstruction feedback to text encoder optimization, we introduce an additional ablation that blocks MSE gradients from reaching LeGO-CLIP while retaining the consistency constraints. This increases FID from 0.039 to 0.053 and reduces R-Precision (Top-3) from 0.888 to 0.859, demonstrating the additional importance of reconstruction feedback to generation-oriented text representation learning.

MethodFID ↓R@Top3 ↑MM Dist ↓
Block MSE gradients to CLIP0.088±.0030.859±.0062.603±.012
LeGO0.039±.0010.888±.0022.382±.010

[Q3]: Discussion of MATE in Related Work

We thank the reviewer for this suggestion. We have added a discussion of MATE in Related Work, clarifying that MATE learns text representations through word-motion alignment, whereas LeGO optimizes them directly using feedback from motion generation. The added text is highlighted in blue in the revised manuscript.

[Q4]: Clarification of FID and R-Precision Claims

We have clarified that LeGO achieves the highest R-Precision among all compared methods on all three benchmarks (HumanML3D, KIT-ML, and SnapMoGen). For FID, LeGO achieves the best results among the compared diffusion-based methods on HumanML3D and SnapMoGen, while remaining competitive on KIT-ML. We have revised Section 4.1 accordingly, the added text is highlighted in blue in the revised manuscript.

[Q5]: Correction of the Projector Training Objective

We thank the reviewer for pointing this out. The sign errors in Eq. (2) were introduced during manuscript preparation. Our intended objective combines InfoNCE loss with cosine similarity loss to maximize matched-pair alignment. We have corrected Eq. (2) as follows:

=-\frac{1}{2N}\sum_{i=1}^{N} \left( \log\frac{\exp(S_{ii}/\tau)}{\sum_j\exp(S_{ij}/\tau)} + \log\frac{\exp(S_{ii}/\tau)}{\sum_j\exp(S_{ji}/\tau)} \right) +\frac{1}{N}\sum_{i=1}^{N}(1-S_{ii}).$$

Thank you for adding the MATE comparison under the keyword-substitution protocol and clarifying how LeGO differs from MATE. The ablation that blocks reconstruction gradients to the text encoder addresses my main concern about the role of the projectors. I also appreciate the correction to Eq. (2) and the more precise FID claim in Section 4.1.

I have one remaining clarification. For the blocked-gradient ablation, the response text reports an FID of 0.053, while both the response table and Table 5 of the revised manuscript report 0.088 ± 0.003. Could you confirm which value is correct? This discrepancy does not change my interpretation of the ablation.

We thank the reviewer for carefully pointing out this inconsistency.

We confirm that the correct FID is 0.088 ± 0.003. This result was already reported in the revised manuscript uploaded five days ago, specifically in Table 5 under the ablation setting "Block MSE gradients to CLIP." The value of 0.053 in our previous response text was incorrect, while both the response table and the revised manuscript report the correct result.

We apologize for this inconsistency and appreciate the reviewer's careful reading.

We sincerely thank you for your constructive feedback and for acknowledging that your concerns have been resolved.

Given that your concerns have now been addressed, we would be very grateful if you could reconsider your overall rating in light of our responses and additional experimental results.

We greatly appreciate your time and thoughtful consideration.

Reviewer_F9uDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LeGO, a generation-oriented text representation learning framework for text-to-motion generation. The method introduces two semantic consistency losses based on pretrained text-motion projectors to jointly optimize the motion generator and CLIP text encoder. Experiments on several benchmarks demonstrate improvements in generation performance.

Critical strengths
  1. The problem is well-motivated, as the widely used text encoder CLIP is not sensitive to fine-grained keywords such as the words describing speed or orientation.
  2. Extensive experiments show that the proposed LeGO model outperforms the other baselines
  3. The paper is well-structured and easy to follow, with helpful figures to explain the setting and idea.
Critical weaknesses
  1. After reading the main paper and the appendix H, I understand the proposed model is effective, but I’m still not sure whether there is a potential evaluation bias or coupling, as the pretrained projectors are trained with text-motion contrastive learning losses similar to T2M evaluators.
    • The original MDM’s FID and R@Top3 R-Precision are 0.489 and 0.749 as shown in Table 1, while from Table 9 in Appendix H, if we apply similar training as in LeGO with TMR or official evaluators, the performance improves significantly. The FID improves to 0.045 and 0.103, and R@Top3 improves to 0.879 and 0.871, where the improvement is similar to the proposed projectors. This raises concerns about whether the large improvements are from similarities between the training supervision and evaluator objectives or not.
    • In the zero-shot evaluation on the 67-dim benchmark and the different evaluator on SnapMoGen section, as the LeGO projectors use 292-dim and 263-dim representation while the corresponding evaluators use a subset of the representation (148-dim and 67-dim), which cannot fully reduce the concern.
  2. The equation 2 has incorrect signs for the two loss terms.
Questions for authors

I acknowledge the effectiveness of the proposed method and appreciate the extensive experiments. However, my main concern is whether the large improvement is partially from the potential evaluation bias due to similar projector training objectives. I am open for further discussion regarding this question.

2 comments

[Q1]: Clarification of Evaluator Coupling

We thank the reviewer for raising this concern. We provide several complementary pieces of evidence to clarify the relationship between our training supervision and evaluation metrics.

First, our projectors are trained from scratch using only the training split, without sharing weights or checkpoints with the official evaluators. As shown in Table 9, both TMR and the official evaluators yield substantial improvements when used as projectors, demonstrating the general effectiveness of our consistency supervision. Notably, if direct evaluator alignment were the primary source of improvement, one might expect the official evaluator to perform best. However, our projectors achieve better results, suggesting that the gains cannot be attributed solely to such alignment.

More importantly, improvements are also observed beyond learned evaluator metrics:

  1. Evaluator-independent physical and human evaluations. Appendix B evaluates fine-grained keyword-conditioned motion control, while Appendix C examines the simultaneous satisfaction of multiple motion constraints under compositional prompts. Both use physical measurements independent of learned evaluators. LeGO improves clockwise/counter-clockwise success from 55.5% to 97.0% and quickly/slowly pairwise success from 49.0% to 95.0%. It also reduces the foot skating ratio from 0.095 to 0.061 (Table 5). The user study (Appendix L) further confirms improvements in motion realism and semantic alignment.

  2. Plug-and-play transferability. Table 3 shows that frozen LeGO-CLIP consistently improves MDM, MoMask, and SALAD using their original training losses, without our consistency constraints. This supports the utility of the learned representations beyond the original LeGO training framework.

Together, these results support that LeGO's improvements extend beyond alignment with a particular learned evaluator, yielding measurable gains in physical controllability and perceptual motion quality.

[Q2]: Correction of the Projector Training Objective

We thank the reviewer for pointing out this issue. Our intended objective combines InfoNCE loss for cross-modal alignment with cosine similarity loss to maximize the similarity of matched text-motion pairs. We have corrected the sign errors in Eq. (2) as follows:

=-\frac{1}{2N}\sum_{i=1}^{N} \left( \log\frac{\exp(S_{ii}/\tau)}{\sum_j\exp(S_{ij}/\tau)} + \log\frac{\exp(S_{ii}/\tau)}{\sum_j\exp(S_{ji}/\tau)} \right) +\frac{1}{N}\sum_{i=1}^{N}(1-S_{ii}).$$

Thank you for the further explanation on evaluation decoupling and the correction of projector loss. I have no further concern regarding these and will keep the positive rating.

Reviewer_DEutOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper is about howt to let text representations better capture the action details for human motion generation. They propose LeGO, jointly train the motion generator and text encoder using feedback on how well generated motion matches the instruction and reference motion. The learned text encoder can be reused in other motion generators by retraining with it. Eavluations support the benefit of learning text representations through generation feedback.

Critical strengths

Bad text representations is a headache for human motion generation, and a lot of motion dataset includes exceptionally bad text prompts, so I can understand the motivation here. The evaluation, especially ablations, show that adapting the text encoder and adding semantic feedback and improve results and work better together. The learned encoder can also improve several motion generation when they are retrained with it. Physical control tests and human judgments also provide related evidence. Currently I can clearly see the reason why this method can be useful, especially it can be used on general human motion generation.

Critical weaknesses

The physical keyword-control tests compare the full LeGO system with the baseline, so the contribution of learned text encoder is not isolated here. The transfer experiments show gains and standard metrics, but if only the encoder is reused, will we have the same improvements in physical control carry over? The metrics do not directly establish that proof. Also, the signs in Equation (2) seem conflict with the stated projector training objective.

Questions for authors

As mentioned in weakness, for the physical keyword-control tests, maybe also evaluate the model retrained with frozen LeGO-CLIP and the consistency-only variant under matched seeds and noise? It would clarify which behavioral gains transfer with the encoder and which depend on the additional consistency losses.

4 comments

[Q1]:Physical control evaluation on the model with frozen LeGO-CLIP.

We thank the reviewer for the valuable suggestion. We agree that testing with frozen LeGO-CLIP can better isolate the contribution of the learned text representations to physical motion control. Below, we provide additional experimental results comparing three models: MDM, MDM + LeGO-CLIP, and LeGO.

Experiment 1: Left vs. Right

ModelleftrightDiffSuccess Rate
MDM+50.3°-15.4°65.7°80.5
MDM+LeGO-CLIP+55.5°-70.9°126.4°82.4
LeGO+64.2°-73.9°138.1°90.4

Experiment 2: Quickly vs. Slowly

ModelquicklyslowlyDiffSpeed ratioPairwise Success Rate
MDM0.491 m/s0.439 m/s0.0521.12×49.0
MDM+LeGO-CLIP0.754 m/s0.331 m/s0.4232.27x76.5
LeGO1.259 m/s0.508 m/s0.7512.48×95.0

Experiment 3: Forward vs. Backward

ModelforwardbackwardDiffSuccess Rate
MDM+0.949-0.6101.55991.6
MDM+LeGO-CLIP+0.931-0.6941.62596.7
LeGO+0.912-0.8481.760100.0

Experiment 4: Counter-clockwise vs. Clockwise

Modelcounter-clockwiseclockwiseDiffSuccess Rate
MDM+36.4+15.5 (×)20.955.5
MDM+LeGO-CLIP+12.6-27.640.273.0
LeGO+32.3-36.268.697.0

Experiment 5: Left Hand vs. Right Hand

Modelleft handright handDiffSuccess Rate
MDM+0.969 m-1.159 m2.128 m100.0
MDM+LeGO-CLIP+0.954m-1.023m1.977m100.0
LeGO+1.071 m-1.083 m2.153 m100.0

Experiment 6: Run vs. Walk

ModelrunwalkDiffSpeed ratioPairwise Success Rate
MDM1.536 m/s0.791 m/s0.7451.94×82.0
MDM+LeGO-CLIP2.029 m/s1.040 m/s0.9891.95x88.0
LeGO2.007 m/s0.850 m/s1.1572.36×92.0

These results demonstrate that the generation-oriented representations learned by LeGO-CLIP can significantly improve downstream motion generation and physical controllability, even when trained solely with MSE loss, without our proposed consistency constraints.

[Q2]: Correction of the Projector Training Objective

We thank the reviewer for pointing out this issue. Our intended objective combines InfoNCE loss for cross-modal alignment with cosine similarity loss to maximize the similarity of matched text-motion pairs. We have corrected the sign errors in Eq. (2) as follows:

=-\frac{1}{2N}\sum_{i=1}^{N} \left( \log\frac{\exp(S_{ii}/\tau)}{\sum_j\exp(S_{ij}/\tau)} + \log\frac{\exp(S_{ii}/\tau)}{\sum_j\exp(S_{ji}/\tau)} \right) +\frac{1}{N}\sum_{i=1}^{N}(1-S_{ii}).$$

Thank you for running additional experiments. I mainly wanted to see whether the better speed and direction control carried over from the learned text encoder itself rather than requiring full LeGO training, and I believe the clear jump over vanilla MDM answers that well. Together with the paper's existing ablations, my main concern is already resolved. I've also checked other reviews and comments, there is no other further concern and I am glad to raise my rating.

If accepted, please also incorporate these additional results into the revised paper.

Thank you for the positive feedback. We will incorporate the additional experimental results into the revised paper.

We sincerely thank you again for your positive feedback and for acknowledging that your main concerns have been resolved.

As you previously mentioned that you would be glad to raise your rating, we would greatly appreciate it if you could update your score accordingly when convenient.

Thank you again for your time and thoughtful evaluation of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussions for AC

We sincerely appreciate the valuable time and effort of the Area Chair and all reviewers. We have carefully addressed all concerns through additional experiments and detailed clarifications of key components.

We are pleased that all three reviewers have explicitly acknowledged that their concerns have been addressed.

For Reviewer DEut, we have provided additional keyword-control experiments using frozen LeGO-CLIP. She/he then expressed clear willingness to raise the rating, i.e., 3-->4. (But it seems that she/he forgot to set the score to 4 immediately).

For Reviewer F9uD, we have clarified the concern regarding evaluation. (current score: 3)

For Reviewer LAaK, for baseline comparisons, we have reproduced MATE (ICLR2026) and added semantic sensitivity analysis with MATE. We also clarify the critical role of mse reconstruction feedback in complementing the two high-level consistency losses. We sincerely look forward to hearing from Reviewer LAaK when they have an opportunity to update her/his rating.

We sincerely thank the reviewers again for their constructive feedback and hope that the AC will take these positive developments into consideration when making the final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.