- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether VLMs can conduct fine-grained evaluation for text-to-motion (T2M) generation. The authors build FineMo, a benchmark of 1,600 base text-motion groups (x 6 data-induced semantic dimensions x 3 tasks = 14,400 questions in total). Evaluating six VLMs, the authors identify three failure modes (depth perception, topology discrimination, temporal resolution).
Critical strengths
- The research question is timely and important. VLMs are increasingly used as judges for generated motion without systematic validation, and fine-grained, diagnosable evaluation of T2M is a real gap left by global similarity metrics such as R-Precision and MM-Dist.
- The benchmark design is well motivated. Its main strengths are single-dimension counterfactual negatives graded by semantic distance, data-driven dimension induction, and a progressive task design.
Critical weaknesses
-
The central claim is questionable. The paper is implemented to evaluate generated motion, but every experiment uses ground-truth mocap + generated counterfactual texts. To be more accurate, the output should be obtained from actual T2M models (e.g., MDM, MLD, MotionDiffuse, KinMo, MoMask). Moreover, existing metrics (R-Precision, MM-Dist, MM-diversity, FID, TMR etc) already provides a comprehensive evaluation on generated motion, VLM's evaluations and human evaluations are independent. Without correlation analysis, the claims of "reliable deployment" and that VLMs fill the gap left by existing metrics remain unverified. If a motion-text specialty model already performs strongly on FineMo, the case for VLM-based evaluation is weakened. The authors acknowledged this in Limitation (1), but since it is the paper’s central claim, we view it as a critical gap rather than a weakness.
-
There are multiple different representations for human motion, such as consecutive SMPL, AMASS, velocity representation, video, motion tokens, joint coordinates, etc. Different VLM or motion LLM take different modalities as inputs (one or more modalities). I admit that motions can be rendered into images, but there are more errors than directly feeding motions into some compatible models. For this paper, if motion-native models do not exhibit the depth or topology failures, the failures are mainly rendering-induced; if they do, the failures lie in semantic alignment. Therefore, the consideration of this pipeline is not completed enough.
Questions for authors
-
Can you evaluate the VLM judges (w/o the adaptive framework) on motions generated by T2M models mentioned above, and report correlation with human ratings? How does this compare with fixed metrics such as R-Precision, MM-Dist, and TMR-based scores?
-
Can you also evaluate motion-language models (e.g., MotionGPT, MotionLLM, MG-MotionLLM) that take motion representations directly as input? The conversion scripts between different modalities can be found under the TMR github repo
2 comments
Weakness 2 & Q2: Models That Take Motion as Input
We agree that testing only the "render, then VLM" pipeline cannot separate rendering-induced failures from semantic-alignment failures. We evaluated motion-input models on FineMo and analyze the results using the criterion the reviewer proposed.
Models and adaptation. Representations are converted with the scripts in the TMR repository. TMR ranks candidate texts by text-motion similarity; MotionGPT and MG-MotionLLM score each candidate by its length-normalized log-likelihood conditioned on motion tokens; MotionLLM (Chen et al., 2024) supports question answering and uses the paper's ranking prompt. To separate input modality from scoring method, we also let Qwen2.5-VL score candidates by likelihood. To separate input modality from the model itself, we include the same VLM with improved rendering: 6 views for depth-related groups, skinned mesh for topology-related groups, 4× FPS for temporal-resolution groups, and the original rendering otherwise (settings as in Tables 9–11).
Ranking accuracy (%, all 1,600 groups, categories as in Table 3):
| Model | Input | Depth | Topology | Temporal | Non-failure-mode | Overall |
|---|---|---|---|---|---|---|
| Qwen2.5-VL, generative | Skeleton rendering | 25 | 40 | 25 | 60 | 41 |
| Qwen2.5-VL, likelihood | Skeleton rendering | 27 | 41 | 26 | 61 | 42 |
| Qwen2.5-VL, improved rendering | Per-category rendering | 42 | 52 | 37 | 60 | 49 |
| Qwen2.5-VL + Adaptive | Adaptive | 53 | 61 | 45 | 61 | 55 |
| TMR | Joint coordinates | 48 | 44 | 31 | 62 | 48 |
| MotionGPT | Motion tokens | 42 | 41 | 26 | 55 | 42 |
| MotionLLM | Motion | 44 | 43 | 27 | 57 | 44 |
| MG-MotionLLM | Motion tokens | 47 | 52 | 30 | 58 | 47 |
TMR and likelihood scoring are invariant to option position. MotionLLM outputs an identical order under all three rotations for 9% of groups; results that count a group as correct only if all rotations are ranked correctly are in the appendix, together with the judgment results. Error correction requires free-form text; among the four models only MotionLLM supports it, with a semantic match rate of 19% versus 31% for the original Qwen2.5-VL.
Training-data overlap. FineMo motions come mainly from AMASS, and the motions of 71% of groups appear in the training data of at least one of the four models. On the 412 groups without overlap (mostly from Motion-X), the ordering of models is unchanged (TMR 46%, original Qwen2.5-VL 40%, Qwen2.5-VL + Adaptive 54%).
TMR's overall accuracy on FineMo (48%) is higher than that of the original VLM (41%) and lower than that of the VLM with the adaptive framework (55%).
Analysis following the reviewer's criterion:
- Depth-related. Motion-input models (42–48%) are well above the skeleton-rendered VLM (25%) and comparable to the same VLM with 6 views (42%). The gap from 25% to about 42% stems from depth information lost in rendering; on top of this, explicit depth maps raise accuracy to 53%, consistent with the two-stage attribution in Section 5.1.
- Topology-related. The same VLM rises from 40% to 52% with skinned-mesh rendering. Among motion-input models, MG-MotionLLM, trained on body-part-level descriptions, reaches 52%, while the others (41–43%) are on par with the original VLM. This failure can occur with either input type.
- Temporal-resolution. Motion-input models score 26–31%, below the same VLM at 4× FPS (37%). The motion tokenizers of MotionGPT and MG-MotionLLM downsample in time and also lose temporal detail. This failure is present with both input types and is not resolved by avoiding rendering.
The sources of failure therefore differ by type. The revision will update the per-failure-mode attribution in Section 5 and extend the FineMo analysis to retrieval models and motion-language models.
We thank the reviewer for recognizing the research question and the benchmark design, and for proposing concrete tests of the central claim. We address both questions with new experiments; due to the length limit, our response is split into two comments.
Weakness 1 & Q1: Correlation with Human Ratings on Generated Motions
We agree that this is the evidence on which the central claim rests and that it was missing from the submission. We therefore added a correlation analysis with human ratings on actual T2M outputs.
Data. We generated 50 motions each with MDM, MotionDiffuse, MLD, and MoMask from HumanML3D test prompts (200 text-motion pairs). To ensure enough samples per semantic dimension, the six dimensions appear in a comparable number of prompts (counted with the decomposition of Appendix A.1). Three annotators independently rated alignment on a 1–5 scale (Krippendorff's α = 0.74); the mean serves as the reference.
Evaluators. Fixed metrics: MM-Dist (text-motion distance from the evaluator of Guo et al.), the sample-level retrieval rank underlying R-Precision (rank of the paired text among 32 candidates, 31 drawn at random from the test set), and TMR text-motion similarity; distances and ranks are negated before computing correlations. VLM judges use the paper's unified input (skeleton, single front view, 15 FPS) without the adaptive framework, with a 1–5 score field added to the judgment prompt. Scoring open-source models by P("Yes") gives similar results (Qwen2.5-VL: 0.39 with P("Yes"), 0.37 with the 1–5 score; see appendix). For reference, we include a re-implementation of VeMo (Ji et al., ICML 2026), a published VLM-based T2M evaluator (Qwen2.5-VL; skinned mesh, 6 views, likelihood scoring, entropy-based view selection), with and without the adaptive framework.
Without the framework, the stronger VLMs (Qwen3.5-Plus, Gemini 2.5 Pro, GPT-5.4) exceed MM-Dist and R-Precision, and their differences from TMR are not significant (largest gap, GPT-5.4: paired bootstrap Δρ = +0.06 [−0.04, 0.16]); LLaVA-1.6 and InternVL2 fall below TMR. The ordering among VLMs matches Table 2. With VeMo-style rendering and scoring, Qwen2.5-VL exceeds TMR (Δρ = +0.10 [0.01, 0.19]); with the adaptive framework added, the gap is +0.19 [0.09, 0.29].
MM-Dist, R-Precision, and TMR output one score per pair and do not localize errors. For VLM evaluators, we run error correction on each dimension involved in the text and take the dimensions reported as erroneous as the localized prediction. Against human-annotated error dimensions, localization F1 ranges from 0.39 (VeMo-Repro) to 0.47 (with the adaptive framework); an LLM that sees only the text and predicts likely error dimensions reaches 0.24.
FID and MM-diversity are distribution-level metrics and cannot be correlated per sample. By mean human rating, the four generators rank MoMask > MLD > MDM > MotionDiffuse, and FID gives the same ranking; four points are too few for a meaningful correlation, so we report the ranking only.
The revised abstract and introduction will state that, without configuration, stronger VLMs correlate with human ratings on par with TMR and additionally output the dimension of each error; deployment claims will be limited to the settings validated above.