acceptodds
Under review as a conference paper at ICLR 2027

The Evaluator Is a Random Variable: Auditing the Fixed Feature Space Behind Text-to-Motion Benchmarks

Abstract

Text-to-motion generation is benchmarked almost entirely through one artifact: a text-motion contrastive feature extractor released in 2022 whose embedding space defines FID, R-precision, matching score, Diversity, and MultiModality on HumanML3D and KIT-ML. Every confidence interval in the literature is computed over generation repeats with this evaluator held fixed, so it omits the evaluator's own training randomness. We treat the evaluator as a random variable. Retraining it from its public recipe with 30 independent seeds on KIT-ML, we score frozen pools of generations from four released state-of-the-art models under every seed and decompose metric variance into evaluator-seed, generation-seed, and batching components. The evaluator seed is the largest component for FID, R-precision, and matching score: for MoMask, the FID standard deviation across seeds is 0.05, seven times the batching noise that published intervals report and larger than the generation-seed component. Among the orderings of our pools under the released checkpoint, every pair whose seed-mean gap is within two seed standard deviations reverses under some seeds, and the gap under the released checkpoint does not predict which; a non-generative baseline that returns the nearest training motion for each test caption overtakes the strongest generator under half of the seeds, an advantage that disappears when duplicate retrieved motions are removed. Inference hyperparameters selected on one evaluator transfer to other seeds with regret no larger than the seed noise, while the identity of the optimum does not. A corruption suite shows that FID ignores a foot drift larger than the standard deviation of its own velocity features, and identity controls show that the feature-extraction route alone can move FID by more than a published state-of-the-art margin. We release the seeded evaluators, the frozen pools, and an evaluator-adjusted confidence interval so that future improvements can be reported against the noise floor of the measurement itself.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
text-to-motion generation, evaluation methodology, benchmark reliability, learned evaluators, FID, R-precision, variance decomposition, reproducibility, human motion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Z4C8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Existing text-to-motion models are quantitatively benchmarked on two datasets, HumanML3D and KIT-ML, using five standard metrics: FID, R-Precision, Matching Score, Diversity, and MultiModality.
  • All of these metrics are computed in the embedding space of a single learned model—the text-motion matching network released with the HumanML3D benchmark.
  • Core Question: How much of a reported difference between two generators is an inherent property of the generators themselves, and how much is merely an artifact of the particular random draw that produced the evaluator?
Critical strengths
  • The paper addresses a highly important and timely topic that the text-to-motion community urgently needs to discuss.
  • It thoroughly investigates and highlights the fundamental flaws and blind spots of the current FID-based evaluation protocol.
Critical weaknesses
  • The authors claim that the evaluator has never been retrained or replaced by the community since its release. This claim requires further verification. Recent works have increasingly adopted alternative evaluators such as TMR (e.g., MoMask++, BiPO, etc.) or MARDM . An analysis extending to these newer evaluators is necessary.

  • It is difficult to be confident that the proposed evaluation setup fully resolves the underlying issues (e.g., seed noise and randomness). The authors should verify whether the evaluated methods (MoMask, BAMM, etc.) maintain consistent performance rankings as the number of evaluator seeds increases (e.g., across 30, 50, or 100 seeds).

  • While existing works routinely include qualitative human evaluations alongside quantitative metrics, this paper lacks any discussion regarding them.

  • A comparative analysis on the Corruption suite and Foot drift test using TMR and MARDM evaluators is needed. If the authors can demonstrate consistent trends across these evaluators as well, I would be inclined to raise my score.

Questions for authors
  • Are there any inherent flaws or limitations in the embedding space of the text-motion matching network caused by text-motion mismatches present within the dataset itself?

  • Regarding "Rank stability with a finite-sample bound": Wouldn't averaging over 20 generation repeats smooth out this variance and yield similar results? It seems necessary to provide a direct comparison between the performance obtained via your methodology and the performance obtained by running the standard 20-repeat evaluation protocol on the released checkpoints yourself.

2 comments

Thank you for the concrete requests.

W1. "Never retrained" and newer evaluators. You are right, and we have corrected the introduction: MARDM retrained the network on its 64-d representation and reports KIT-ML under it, and SnapMoGen/MoMask++ use a TMR-style evaluator. Each is still one trained draw, so our question applies unchanged. We rescored every pool under both released MARDM KIT evaluators (64-d Guo architecture; CLIP text). Our conversion reproduces MARDM's published MoMask number (generation seed 0: FID 0.427 vs 0.411; R-precision 0.367 vs 0.369). FID order, averaged over generation seeds:

  • Guo (released): MoMask 0.193 < retrieval 0.274 < BAMM 0.293 < MDM 0.438 < T2M-GPT 0.707
  • MARDM 64-d: retrieval 0.253 < MoMask 0.425 < MDM 0.589 < T2M-GPT 0.653 < BAMM 0.753
  • MARDM CLIP: retrieval 0.0054 < T2M-GPT 0.0085 < MoMask 0.0127 < MDM 0.0146 < BAMM 0.0169
  • TMR: retrieval 0.028 < T2M-GPT 0.034 < MDM 0.044 < MoMask 0.068 < BAMM 0.077 The retrieval baseline is first and BAMM last under every independently trained evaluator. Caveats: BAMM's KIT checkpoint is an untested release (8ZUj W2); T2M-GPT and retrieval produce shorter motions (68 vs 76 frames), which may contribute to their gains; and Meng et al. report the published ordering preserved under their evaluator, using reported numbers rather than released checkpoints. TMR's KIT model reads SMPL-skeleton motion at 20 fps, so we convert our KIT motions (joint correspondence, resampling, TMR's own feature code), reference and generated sets alike. Converted ground-truth motions reach R-precision top-1 0.36 under TMR (TMR reports 0.50 on its own KIT test), a caveat on the TMR numbers.

W2. Do rankings stabilize with more seeds? We trained 70 more evaluator seeds (100 in total). Statistics of the recipe converge: the evaluator-seed SD of MoMask FID is 0.042 at 5 seeds, 0.047 at 20 and 0.048 at 100 (generation seed 0), and the ranking of seed-averaged scores matches the 100-seed ranking in 75% of 5-seed subsets, 93% at 20 and over 99% at 50. Disagreement between individual evaluators does not go away: a single evaluator ranks the retrieval baseline above MoMask with probability 0.61 at every subset size; only the interval narrows ([0.30, 0.90] at 10 seeds, [0.52, 0.70] at 50).

W3. Human evaluation. We agree it matters for judging generators; our claim is narrower: we measure the variance of an automated metric, which a user study cannot supply. Prior work reports weak sample-level metric-human agreement (Voas et al., 2023; MotionCritic, Wang et al., 2025); we add that the metric side of any such correlation is itself a random variable. We added this to the related work and limitations; a user study is not feasible in the discussion window.

W4. Corruption suite under other evaluators. This was your stated condition; done for MARDM and TMR:

corruption (largest level)Guo releasedMARDM 64-dMARDM CLIPTMR
identity control0.0110.0070.0001<0.0001
foot drift, 8% of body height0.0200.0100.00010.0001
foot-only jitter, 8%6.31.430.01740.099
all-joint jitter, 4%68.556.60.2320.62
caption permutation, 100% (FID)0.0240.0250.00030.0006
caption permutation, 100% (R-precision top-1)0.030.030.020.03

The trends are the same in all four: smooth foot drift is invisible even to the 64-d evaluator, whose input is joint positions only, while jitter on the same four joints is detected; FID cannot see caption-motion mismatch. Under TMR, foot jitter scores about 1000 times foot drift (0.099 vs 0.0001), so the skeleton conversion preserves foot motion.

Q1. Flaws from text-motion mismatch in the data. Yes, and the retrieval result may be one symptom: 34% of KIT-ML test captions appear verbatim in training, often attached to different motions, and the contrastive loss can treat such duplicates in a batch as negatives. The duplicate-free control establishes that the retrieval advantage depends on caption-frequency weighting, though not the mechanism. FID cannot detect a mismatch by construction (caption permutation leaves it at the identity level), so text alignment rests entirely on R-precision, where the evaluator seed accounts for 65-73% of the variance.

Q2. Would 20 generation repeats smooth this out? No: averaging over repeats removes generation and batching noise, but every repeat uses the same evaluator. Over 20 repeats (10 for MDM), for MoMask the published-style half-width under the released checkpoint shrinks to 0.007, while the 20-repeat mean still varies across evaluator seeds with SD 0.047. The same holds for every generator (half-width vs evaluator-seed SD): BAMM 0.013 vs 0.066, T2M-GPT 0.033 vs 0.114, MDM 0.030 vs 0.075. Our 20-repeat released-checkpoint means vs the published values: MoMask 0.193 vs 0.204, MDM 0.438 vs 0.497, BAMM 0.293 vs 0.183, T2M-GPT 0.707 vs 0.514 (8ZUj W2 covers the last two).

Thank you for the extensive experiment. However, I believe that for the proposed protocol to make a substantial contribution to the paper, the following two issues must be addressed.

  1. As you mentioned, if part of the variance, as in the 34% of the test captions, is due to data defects, this issue may be limited to KIT-ML. Therefore, validation on another dataset is required.
  2. In that regard, an evaluation of HumanML3D is essential. However, it is difficult to accept an answer given to another reviewer (QoUU). AMASS and SMPL are available immediately upon license registration. Unless validated by key benchmarks, it is difficult to consider the conclusions generalizable.

Therefore, I will maintain my current score.

Reviewer_8ZUjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how text-to-motion scores change when the evaluator is retrained. The authors train 30 evaluators on KIT-ML, keeping the movement encoder fixed, and score the same generated motions with each evaluator. They examine score variation, ranking changes, hyperparameter selection, and responses to motion corruption.

Critical strengths
  1. The paper measures a source of variation that standard evaluation usually leaves out. This is useful when comparing methods with small score differences.
  2. Scoring the same motion pools across evaluators is a good experimental design. It separates evaluator changes from changes in generated content.
  3. The authors include useful controls and clearly report the limits of their experiments, including results where rankings and hyperparameter choices remain stable.
Critical weaknesses
  1. The paper adds useful information, but I do not think the current evidence is enough for acceptance. Among the tested generators, FID rankings change under only a few evaluator seeds, and selected hyperparameters generally transfer well to other evaluators. The effect on model comparison is therefore limited in these experiments.
  2. All experiments use KIT-ML, and the full variance analysis covers only MoMask. The BAMM and T2M-GPT pools also do not match their published scores. These limits make it difficult to judge how widely the findings apply.
  3. The statistical reporting needs more detail. I could not find the definition of the evaluator-adjusted confidence interval promised in the abstract. It is also unclear how the analysis in Table 3 separates the effects of evaluator seeds, generation seeds, batching, and their interactions.
Questions for authors
  1. How is the evaluator-adjusted confidence interval computed, and what quantity does it estimate?
  2. How does Table 3 account for interactions between the sources of variation?
  3. What explains the BAMM and T2M-GPT reproduction gaps? Do the main ranking results hold when each generator is evaluated with multiple generation seeds?
1 comment

Thank you for the precise questions; two of them exposed gaps we have fixed.

W1. "The effect on model comparison is limited." We agree, and the 100-seed results confirm it: among our four generators, orderings reverse under at most 5 of 100 evaluator seeds, and selected hyperparameters mostly transfer. Our claim is about the resolution of the measurement, not the frequency of reversals among these pools. The evaluator-seed SD of a FID difference between two generators is 0.055 to 0.155 (0.055 for MoMask-BAMM), so a gap reported under one evaluator carries an evaluator-adjusted 95% interval of at least about plus or minus 0.11 (Q1), against plus or minus 0.012 for the published-style interval. Our MoMask-BAMM gap (0.132) exceeds it and is resolved; the published gap between the two best KIT-ML methods (BAMM 0.183 vs MoMask 0.204, 0.021) is well inside it and is not. The one frequent reversal involves the retrieval baseline (61 of 100 seeds), whose advantage the paper ties to duplicated test captions.

W2. KIT only; full variance only for MoMask; BAMM and T2M-GPT gaps. All four generators now have 20 generation seeds (10 for MDM; see QoUU W2). The reproduction gaps are partly explained, now documented in Appendix A.8:

  • BAMM has no KIT checkpoint in its README; the only released KIT weights were posted by the author in a GitHub issue, described as untested, as latest.tar. The bundle's own log reports FID 0.187 with the author's length-predicting KIT variant and 0.200 with ground-truth length; we use ground-truth length (as for every pool) and one caption and crop per test motion. This explains part of the gap; our 20-repeat mean remains 0.09 above the bundle's own 0.200, which we cannot fully account for.
  • T2M-GPT's KIT weights also come from a GitHub issue; the shipped checkpoint's own log reports FID 0.383 at its best-FID iteration and 0.839 at its final iteration on the validation protocol, and the model chooses its own length (mean 68 frames vs 76 for the test crops).
  • Over 20 generation repeats, our released-checkpoint means are MoMask 0.193 (published 0.204), MDM 0.438 over 10 repeats (published 0.497), BAMM 0.293 (0.183) and T2M-GPT 0.707 (0.514). The two documented checkpoints match or improve on their published values; the two undocumented ones remain above theirs.

Q1. Evaluator-adjusted interval. For a metric difference Delta between two methods (new paragraph in Section 3; Table 10):

  • Recipe-level interval (when both methods are scored under the released seeds): mean over seeds of Delta plus or minus t_{N-1} s/sqrt(N), where s is the seed SD of Delta. It estimates the gap averaged over evaluators drawn from the public training recipe, the quantity a ranking claim implicitly refers to.
  • Single-checkpoint interval (for a gap reported under one evaluator, e.g. the released one): Delta plus or minus 1.96 sqrt(s_E^2 + s_G^2/n + s_B^2/n), with s_E the evaluator-seed SD of the difference and s_G, s_B its generation and batching SDs over n repeats. The published interval keeps only the last two terms. Examples (FID, released-checkpoint gap; single-checkpoint interval; published-style half-widths are 0.012-0.049): MoMask-BAMM -0.132, [-0.240, -0.023]; BAMM-MDM -0.154, [-0.372, 0.064]; T2M-GPT-MDM 0.281, [-0.002, 0.564]; MoMask-retrieval -0.089, [-0.198, 0.020]. Our scoring code computes both.

Q2. Interactions in Table 3. Table 3's estimator folds interactions into the level means (now stated in Section 3). The new Table 4 is a three-way crossed random-effects ANOVA (expected mean squares) with the three two-way interactions (one observation per cell, so the three-way interaction is part of the residual), on 100 evaluator seeds x 20 generation seeds x 5 batching repeats. For MoMask FID the components are: evaluator 0.047 (80% of variance), generation 0.019 (13%), evaluator x generation 0.011 (4%), and each remaining term at or below 0.005 (1% or less). R-precision top-1 and matching score: evaluator 73% and 92%. The previous estimator (SD of level means) gives nearly identical evaluator values (0.047 for MoMask FID), so no conclusion changes.

Q3. Reproduction gaps; rankings with multiple generation seeds. See W2. On generation-seed-averaged scores the released evaluator orders the pools MoMask 0.193 < retrieval 0.274 < BAMM 0.293 < MDM 0.438 < T2M-GPT 0.707. Across 100 evaluator seeds, MoMask and retrieval swap under 60 (61 with generation seed 0 alone, as in W1), BAMM and MDM under 5, BAMM and retrieval under 2, T2M-GPT and MDM under 1; no other pair ever swaps.

Reviewer_QoUUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies a problem that is often ignored in text-to-motion evaluation: how the evaluator’s own training randomness affects model performance. The authors keep the generated motions fixed and score them with 30 evaluators trained with different random seeds. They find that the scores vary across evaluators and that some model rankings change. These results suggest that small reported gains may depend on which evaluator is used.

Critical strengths
  1. This paper highlights an interesting, important, and often-overlooked issue in text-to-motion evaluation. The evaluator’s own training randomness can affect scores and some model rankings. This is important when a paper reports a small gain using only one evaluator.

  2. To support the findings, this paper presents many experiments. The authors score fixed motion pools with 30 retrained evaluators and study score variation and ranking changes. The experiments show that changing the evaluator seed can change scores and some model rankings.

Critical weaknesses
  1. The experiments only cover KIT-ML and do not test HumanML3D, a main text-to-motion benchmark. The authors explain that they could not obtain the required body models before the deadline. However, it is still unclear whether the same findings hold for HumanML3D. Adding these experiments would strengthen the conclusions.

  2. This comparison is tested only on MoMask. As the authors acknowledge in Section 6, the generation-noise estimate uses only five seeds and has a wide confidence interval. Testing more seeds and more models would strengthen the claim that evaluator randomness is the largest source of variation.

Questions for authors

See the weaknesses.

The authors' findings are interesting, and the experiments support the main observations. However, more evidence is needed to fully support the conclusions. I would be willing to raise my score if the authors address my concerns.

2 comments

Thank you for the careful summary and for stating what would change your assessment.

W1. HumanML3D. We could not add HumanML3D in the discussion window: rebuilding it requires the AMASS motion data and SMPL body models, which we did not obtain in time. We therefore cannot say whether the findings hold there, and the paper's limitation stands: KIT-ML's small test set (734 motions) likely inflates evaluator variance relative to HumanML3D, so every conclusion is stated for KIT-ML. We intend to add the HumanML3D audit to the final version. Within the window we extended generality along the other axes the reviewers raised: three independently trained evaluator families (MARDM's two networks and TMR, on every pool and corruption) and all four generators with full variance decompositions.

W2. Only MoMask; five generation seeds; wide interval. We now decompose variance for all four generators (Table 4 of the revision), with 20 generation seeds each (10 for MDM) and 100 evaluator seeds:

generatorevaluator-seed SD of FID [95% CI]generation-seed SD [95% CI]evaluator share
MoMask0.047 [0.041, 0.054]0.019 [0.015, 0.028]80%
BAMM0.066 [0.058, 0.077]0.029 [0.022, 0.043]75%
MDM0.075 [0.066, 0.087]0.052 [0.036, 0.096]61%
T2M-GPT0.114 [0.100, 0.133]0.080 [0.061, 0.117]59%

The evaluator seed has the largest estimate for all four generators. For MoMask and BAMM the two intervals do not overlap; for MDM and T2M-GPT they do, so there the evaluator component is at least comparable to generation noise. The analysis is a three-way random-effects ANOVA (evaluator x generation x batching, with all two-way interactions); interactions account for 6% of FID variance for MoMask, BAMM and MDM and 11% for T2M-GPT, so the comparison does not depend on how they are handled. For R-precision top-1 the evaluator share is 65-73% and for matching score 83-93% across generators.

Thank you for the authors' response. It has addressed some of my concerns. However, I consider the HumanML3D results important, and the authors have indicated that they cannot provide them at this time. Therefore, I will keep my original rating.

Comments on the paper

1 comment

We thank the reviewers. All three asked whether the findings extend beyond one evaluator recipe, one generator, and one dataset, and two asked how the evaluator-adjusted interval is defined. We ran the following during the discussion period; all are in the revised PDF (changes in blue; new Section 4.6 and Appendices A.7-A.9).

  1. A second, independently trained evaluator. We rescored every pool and the full corruption suite under the two released MARDM KIT-ML evaluators (Meng et al., CVPR 2025; a retrained 64-d Guo-architecture network and a CLIP-text contrastive network). Our conversion reproduces MARDM's published MoMask score under that evaluator (generation seed 0: FID 0.427 vs 0.411, R-precision top-1 0.367 vs 0.369). The corruption findings replicate: foot drift at 8% of body height moves FID from 0.007 (identity) to 0.010, while foot-only jitter at the same amplitude scores 1.43; FID stays within 0.02 of identity under full caption permutation while R-precision falls from 0.38 to 0.03. We also scored everything under the released TMR KIT-ML evaluator (Petrovich et al., ICCV 2023), through a skeleton conversion described in our response to Z4C8; the same trends hold, and the retrieval baseline ranks first under all three independently trained evaluators.
  2. Variance decomposition for all four generators, with 20 generation seeds each (10 for MDM, the slowest sampler) and a three-way random-effects ANOVA with all two-way interactions (Table 4). The evaluator seed is the largest estimated component of FID for every generator: MoMask SD 0.047 (95% CI [0.041, 0.054], 80% of variance), BAMM 0.066 ([0.058, 0.077], 75%), MDM 0.075 ([0.066, 0.087], 61%), T2M-GPT 0.114 ([0.100, 0.133], 59%). For MoMask and BAMM this interval lies entirely above the generation-seed interval; for MDM and T2M-GPT the two overlap. Interactions account for 6% (11% for T2M-GPT).
  3. 100 evaluator seeds instead of 30. The 70 new seeds are indistinguishable from the original 30 (mean MoMask FID 0.263 under the original seeds vs 0.259 under the new ones; Mann-Whitney p = 0.72). With 100 seeds, the retrieval baseline beats MoMask (generation seed 0) under 61 of 100 evaluators (15 of the original 30), and averaged over the recipe its FID is lower by 0.015 (95% CI [0.004, 0.026]).
  4. 20 generation repeats per generator under the standard protocol (10 for MDM). Averaging over them narrows the published-style interval for MoMask to plus or minus 0.007, while the 20-repeat mean still varies across evaluator seeds with SD 0.047.
  5. A precise definition of the evaluator-adjusted interval (new paragraph in Section 3), with intervals for every pair of methods (Table 10).
  6. Corrections. The introduction overstated that the evaluator "has never been retrained" and that MARDM loads the released file. MARDM retrained the evaluator once on its own representation, and SnapMoGen/MoMask++ adopted a TMR-style evaluator. We now say so, and we note that each such evaluator is itself one draw from its recipe, so the audit applies to it as well.
  7. HumanML3D could not be added within the discussion window (see our response to QoUU).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.