- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- Existing text-to-motion models are quantitatively benchmarked on two datasets, HumanML3D and KIT-ML, using five standard metrics: FID, R-Precision, Matching Score, Diversity, and MultiModality.
- All of these metrics are computed in the embedding space of a single learned model—the text-motion matching network released with the HumanML3D benchmark.
- Core Question: How much of a reported difference between two generators is an inherent property of the generators themselves, and how much is merely an artifact of the particular random draw that produced the evaluator?
Critical strengths
- The paper addresses a highly important and timely topic that the text-to-motion community urgently needs to discuss.
- It thoroughly investigates and highlights the fundamental flaws and blind spots of the current FID-based evaluation protocol.
Critical weaknesses
-
The authors claim that the evaluator has never been retrained or replaced by the community since its release. This claim requires further verification. Recent works have increasingly adopted alternative evaluators such as TMR (e.g., MoMask++, BiPO, etc.) or MARDM . An analysis extending to these newer evaluators is necessary.
-
It is difficult to be confident that the proposed evaluation setup fully resolves the underlying issues (e.g., seed noise and randomness). The authors should verify whether the evaluated methods (MoMask, BAMM, etc.) maintain consistent performance rankings as the number of evaluator seeds increases (e.g., across 30, 50, or 100 seeds).
-
While existing works routinely include qualitative human evaluations alongside quantitative metrics, this paper lacks any discussion regarding them.
-
A comparative analysis on the Corruption suite and Foot drift test using TMR and MARDM evaluators is needed. If the authors can demonstrate consistent trends across these evaluators as well, I would be inclined to raise my score.
Questions for authors
-
Are there any inherent flaws or limitations in the embedding space of the text-motion matching network caused by text-motion mismatches present within the dataset itself?
-
Regarding "Rank stability with a finite-sample bound": Wouldn't averaging over 20 generation repeats smooth out this variance and yield similar results? It seems necessary to provide a direct comparison between the performance obtained via your methodology and the performance obtained by running the standard 20-repeat evaluation protocol on the released checkpoints yourself.
2 comments
Thank you for the extensive experiment. However, I believe that for the proposed protocol to make a substantial contribution to the paper, the following two issues must be addressed.
- As you mentioned, if part of the variance, as in the 34% of the test captions, is due to data defects, this issue may be limited to KIT-ML. Therefore, validation on another dataset is required.
- In that regard, an evaluation of HumanML3D is essential. However, it is difficult to accept an answer given to another reviewer (QoUU). AMASS and SMPL are available immediately upon license registration. Unless validated by key benchmarks, it is difficult to consider the conclusions generalizable.
Therefore, I will maintain my current score.
Thank you for the concrete requests.
W1. "Never retrained" and newer evaluators. You are right, and we have corrected the introduction: MARDM retrained the network on its 64-d representation and reports KIT-ML under it, and SnapMoGen/MoMask++ use a TMR-style evaluator. Each is still one trained draw, so our question applies unchanged. We rescored every pool under both released MARDM KIT evaluators (64-d Guo architecture; CLIP text). Our conversion reproduces MARDM's published MoMask number (generation seed 0: FID 0.427 vs 0.411; R-precision 0.367 vs 0.369). FID order, averaged over generation seeds:
W2. Do rankings stabilize with more seeds? We trained 70 more evaluator seeds (100 in total). Statistics of the recipe converge: the evaluator-seed SD of MoMask FID is 0.042 at 5 seeds, 0.047 at 20 and 0.048 at 100 (generation seed 0), and the ranking of seed-averaged scores matches the 100-seed ranking in 75% of 5-seed subsets, 93% at 20 and over 99% at 50. Disagreement between individual evaluators does not go away: a single evaluator ranks the retrieval baseline above MoMask with probability 0.61 at every subset size; only the interval narrows ([0.30, 0.90] at 10 seeds, [0.52, 0.70] at 50).
W3. Human evaluation. We agree it matters for judging generators; our claim is narrower: we measure the variance of an automated metric, which a user study cannot supply. Prior work reports weak sample-level metric-human agreement (Voas et al., 2023; MotionCritic, Wang et al., 2025); we add that the metric side of any such correlation is itself a random variable. We added this to the related work and limitations; a user study is not feasible in the discussion window.
W4. Corruption suite under other evaluators. This was your stated condition; done for MARDM and TMR:
The trends are the same in all four: smooth foot drift is invisible even to the 64-d evaluator, whose input is joint positions only, while jitter on the same four joints is detected; FID cannot see caption-motion mismatch. Under TMR, foot jitter scores about 1000 times foot drift (0.099 vs 0.0001), so the skeleton conversion preserves foot motion.
Q1. Flaws from text-motion mismatch in the data. Yes, and the retrieval result may be one symptom: 34% of KIT-ML test captions appear verbatim in training, often attached to different motions, and the contrastive loss can treat such duplicates in a batch as negatives. The duplicate-free control establishes that the retrieval advantage depends on caption-frequency weighting, though not the mechanism. FID cannot detect a mismatch by construction (caption permutation leaves it at the identity level), so text alignment rests entirely on R-precision, where the evaluator seed accounts for 65-73% of the variance.
Q2. Would 20 generation repeats smooth this out? No: averaging over repeats removes generation and batching noise, but every repeat uses the same evaluator. Over 20 repeats (10 for MDM), for MoMask the published-style half-width under the released checkpoint shrinks to 0.007, while the 20-repeat mean still varies across evaluator seeds with SD 0.047. The same holds for every generator (half-width vs evaluator-seed SD): BAMM 0.013 vs 0.066, T2M-GPT 0.033 vs 0.114, MDM 0.030 vs 0.075. Our 20-repeat released-checkpoint means vs the published values: MoMask 0.193 vs 0.204, MDM 0.438 vs 0.497, BAMM 0.293 vs 0.183, T2M-GPT 0.707 vs 0.514 (8ZUj W2 covers the last two).