acceptodds
Under review as a conference paper at ICLR 2027

Can VLMs Judge Motion? From Capability Diagnosis to Reliable Deployment

Abstract

Mainstream evaluation metrics for text-driven human motion generation (T2M) compute global similarity scores, offering no fine-grained assessment of textmotion alignment. Vision-language models (VLMs), with their capacity for multimodal semantic understanding, are a natural candidate to fill this gap. However, 3D human motion involves spatial depth, skeletal topology, and temporal continuity that lie beyond typical 2D visual reasoning, leaving VLM reliability in this domain unverified—and deploying an unverified evaluator risks producing misleading conclusions in practice. To investigate this, we present a four-part study. (1) We construct FineMo (Fine-Grained Motion Semantic Alignment Benchmark), a diagnostic benchmark comprising 14,400 questions across six data-driven semantic dimensions, three alignment levels, and three progressive task types. (2) We evaluate six mainstream VLMs on FineMo, revealing three failure modes that persist consistently across all models and tasks: Depth Perception Failure, Topology Discrimination Failure, and Temporal Resolution Failure. (3) Tracing each failure, we identify two root causes—rendering-stage information loss and model-stage processing limitation—each verified through controlled experiments. (4) Building on this analysis, we propose an adaptive evaluation framework that diagnoses failure modes from text semantics and activates targeted interventions accordingly, achieving 57% overall accuracy—16 points above baseline and 9 above blanket intervention—without modifying the model. Together, these contributions provide a systematic characterization of VLM capability boundaries in motion evaluation and an evidence-based framework for their reliable deployment.

Primary area:
generative models
Keywords:
Motion Generation, VLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tYckOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether VLMs can conduct fine-grained evaluation for text-to-motion (T2M) generation. The authors build FineMo, a benchmark of 1,600 base text-motion groups (x 6 data-induced semantic dimensions x 3 tasks = 14,400 questions in total). Evaluating six VLMs, the authors identify three failure modes (depth perception, topology discrimination, temporal resolution).

Critical strengths
  1. The research question is timely and important. VLMs are increasingly used as judges for generated motion without systematic validation, and fine-grained, diagnosable evaluation of T2M is a real gap left by global similarity metrics such as R-Precision and MM-Dist.
  2. The benchmark design is well motivated. Its main strengths are single-dimension counterfactual negatives graded by semantic distance, data-driven dimension induction, and a progressive task design.
Critical weaknesses
  1. The central claim is questionable. The paper is implemented to evaluate generated motion, but every experiment uses ground-truth mocap + generated counterfactual texts. To be more accurate, the output should be obtained from actual T2M models (e.g., MDM, MLD, MotionDiffuse, KinMo, MoMask). Moreover, existing metrics (R-Precision, MM-Dist, MM-diversity, FID, TMR etc) already provides a comprehensive evaluation on generated motion, VLM's evaluations and human evaluations are independent. Without correlation analysis, the claims of "reliable deployment" and that VLMs fill the gap left by existing metrics remain unverified. If a motion-text specialty model already performs strongly on FineMo, the case for VLM-based evaluation is weakened. The authors acknowledged this in Limitation (1), but since it is the paper’s central claim, we view it as a critical gap rather than a weakness.

  2. There are multiple different representations for human motion, such as consecutive SMPL, AMASS, velocity representation, video, motion tokens, joint coordinates, etc. Different VLM or motion LLM take different modalities as inputs (one or more modalities). I admit that motions can be rendered into images, but there are more errors than directly feeding motions into some compatible models. For this paper, if motion-native models do not exhibit the depth or topology failures, the failures are mainly rendering-induced; if they do, the failures lie in semantic alignment. Therefore, the consideration of this pipeline is not completed enough.

Questions for authors
  1. Can you evaluate the VLM judges (w/o the adaptive framework) on motions generated by T2M models mentioned above, and report correlation with human ratings? How does this compare with fixed metrics such as R-Precision, MM-Dist, and TMR-based scores?

  2. Can you also evaluate motion-language models (e.g., MotionGPT, MotionLLM, MG-MotionLLM) that take motion representations directly as input? The conversion scripts between different modalities can be found under the TMR github repo

2 comments

We thank the reviewer for recognizing the research question and the benchmark design, and for proposing concrete tests of the central claim. We address both questions with new experiments; due to the length limit, our response is split into two comments.

Weakness 1 & Q1: Correlation with Human Ratings on Generated Motions

We agree that this is the evidence on which the central claim rests and that it was missing from the submission. We therefore added a correlation analysis with human ratings on actual T2M outputs.

Data. We generated 50 motions each with MDM, MotionDiffuse, MLD, and MoMask from HumanML3D test prompts (200 text-motion pairs). To ensure enough samples per semantic dimension, the six dimensions appear in a comparable number of prompts (counted with the decomposition of Appendix A.1). Three annotators independently rated alignment on a 1–5 scale (Krippendorff's α = 0.74); the mean serves as the reference.

Evaluators. Fixed metrics: MM-Dist (text-motion distance from the evaluator of Guo et al.), the sample-level retrieval rank underlying R-Precision (rank of the paired text among 32 candidates, 31 drawn at random from the test set), and TMR text-motion similarity; distances and ranks are negated before computing correlations. VLM judges use the paper's unified input (skeleton, single front view, 15 FPS) without the adaptive framework, with a 1–5 score field added to the judgment prompt. Scoring open-source models by P("Yes") gives similar results (Qwen2.5-VL: 0.39 with P("Yes"), 0.37 with the 1–5 score; see appendix). For reference, we include a re-implementation of VeMo (Ji et al., ICML 2026), a published VLM-based T2M evaluator (Qwen2.5-VL; skinned mesh, 6 views, likelihood scoring, entropy-based view selection), with and without the adaptive framework.

EvaluatorSpearman ρ [95% CI]Kendall τ
MM-Dist0.24 [0.11, 0.37]0.17
R-Precision (retrieval rank)0.21 [0.08, 0.34]0.15
TMR similarity0.38 [0.26, 0.49]0.28
LLaVA-1.60.20 [0.07, 0.33]0.14
InternVL20.29 [0.16, 0.41]0.21
Qwen2.5-VL0.37 [0.25, 0.48]0.27
Qwen3.5-Plus0.40 [0.28, 0.51]0.30
Gemini 2.5 Pro0.43 [0.32, 0.53]0.32
GPT-5.40.44 [0.33, 0.54]0.33
VeMo-Repro0.48 [0.37, 0.58]0.37
VeMo-Repro + Adaptive0.57 [0.47, 0.66]0.45

Without the framework, the stronger VLMs (Qwen3.5-Plus, Gemini 2.5 Pro, GPT-5.4) exceed MM-Dist and R-Precision, and their differences from TMR are not significant (largest gap, GPT-5.4: paired bootstrap Δρ = +0.06 [−0.04, 0.16]); LLaVA-1.6 and InternVL2 fall below TMR. The ordering among VLMs matches Table 2. With VeMo-style rendering and scoring, Qwen2.5-VL exceeds TMR (Δρ = +0.10 [0.01, 0.19]); with the adaptive framework added, the gap is +0.19 [0.09, 0.29].

MM-Dist, R-Precision, and TMR output one score per pair and do not localize errors. For VLM evaluators, we run error correction on each dimension involved in the text and take the dimensions reported as erroneous as the localized prediction. Against human-annotated error dimensions, localization F1 ranges from 0.39 (VeMo-Repro) to 0.47 (with the adaptive framework); an LLM that sees only the text and predicts likely error dimensions reaches 0.24.

FID and MM-diversity are distribution-level metrics and cannot be correlated per sample. By mean human rating, the four generators rank MoMask > MLD > MDM > MotionDiffuse, and FID gives the same ranking; four points are too few for a meaningful correlation, so we report the ranking only.

The revised abstract and introduction will state that, without configuration, stronger VLMs correlate with human ratings on par with TMR and additionally output the dimension of each error; deployment claims will be limited to the settings validated above.

Weakness 2 & Q2: Models That Take Motion as Input

We agree that testing only the "render, then VLM" pipeline cannot separate rendering-induced failures from semantic-alignment failures. We evaluated motion-input models on FineMo and analyze the results using the criterion the reviewer proposed.

Models and adaptation. Representations are converted with the scripts in the TMR repository. TMR ranks candidate texts by text-motion similarity; MotionGPT and MG-MotionLLM score each candidate by its length-normalized log-likelihood conditioned on motion tokens; MotionLLM (Chen et al., 2024) supports question answering and uses the paper's ranking prompt. To separate input modality from scoring method, we also let Qwen2.5-VL score candidates by likelihood. To separate input modality from the model itself, we include the same VLM with improved rendering: 6 views for depth-related groups, skinned mesh for topology-related groups, 4× FPS for temporal-resolution groups, and the original rendering otherwise (settings as in Tables 9–11).

Ranking accuracy (%, all 1,600 groups, categories as in Table 3):

ModelInputDepthTopologyTemporalNon-failure-modeOverall
Qwen2.5-VL, generativeSkeleton rendering2540256041
Qwen2.5-VL, likelihoodSkeleton rendering2741266142
Qwen2.5-VL, improved renderingPer-category rendering4252376049
Qwen2.5-VL + AdaptiveAdaptive5361456155
TMRJoint coordinates4844316248
MotionGPTMotion tokens4241265542
MotionLLMMotion4443275744
MG-MotionLLMMotion tokens4752305847

TMR and likelihood scoring are invariant to option position. MotionLLM outputs an identical order under all three rotations for 9% of groups; results that count a group as correct only if all rotations are ranked correctly are in the appendix, together with the judgment results. Error correction requires free-form text; among the four models only MotionLLM supports it, with a semantic match rate of 19% versus 31% for the original Qwen2.5-VL.

Training-data overlap. FineMo motions come mainly from AMASS, and the motions of 71% of groups appear in the training data of at least one of the four models. On the 412 groups without overlap (mostly from Motion-X), the ordering of models is unchanged (TMR 46%, original Qwen2.5-VL 40%, Qwen2.5-VL + Adaptive 54%).

TMR's overall accuracy on FineMo (48%) is higher than that of the original VLM (41%) and lower than that of the VLM with the adaptive framework (55%).

Analysis following the reviewer's criterion:

  • Depth-related. Motion-input models (42–48%) are well above the skeleton-rendered VLM (25%) and comparable to the same VLM with 6 views (42%). The gap from 25% to about 42% stems from depth information lost in rendering; on top of this, explicit depth maps raise accuracy to 53%, consistent with the two-stage attribution in Section 5.1.
  • Topology-related. The same VLM rises from 40% to 52% with skinned-mesh rendering. Among motion-input models, MG-MotionLLM, trained on body-part-level descriptions, reaches 52%, while the others (41–43%) are on par with the original VLM. This failure can occur with either input type.
  • Temporal-resolution. Motion-input models score 26–31%, below the same VLM at 4× FPS (37%). The motion tokenizers of MotionGPT and MG-MotionLLM downsample in time and also lose temporal detail. This failure is present with both input types and is not resolved by avoiding rendering.

The sources of failure therefore differ by type. The revision will update the per-failure-mode attribution in Section 5 and extend the FineMo analysis to retrieval models and motion-language models.

Reviewer_LQ6aOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether vision-language models can reliably evaluate fine-grained semantic alignment between text and 3D human motion. The authors introduce FineMo, a benchmark constructed from 1,600 motion groups drawn from four motion datasets. It covers six semantic dimensions and three tasks: binary judgment, ranking, and error correction. Experiments with six VLMs identify three recurring failure modes involving depth perception, skeletal-topology discrimination, and temporal resolution. The paper then evaluates interventions based on multi-view rendering, depth maps, body-part annotations, and task-specific keyframes, and proposes an LLM-routed adaptive evaluation framework. Benchmark quality is assessed using a 300-group human-annotated subset, and the intervention trends are tested on Qwen2.5-VL and InternVL2.

Critical strengths
  1. A useful diagnostic benchmark. FineMo separates motion-text alignment into six interpretable dimensions and combines graded negatives with judgment, ranking, and error-correction tasks. The consistent difficulty patterns across six VLMs reveal fine-grained limitations that conventional aggregate T2M metrics cannot expose.
  2. A systematic intervention study. The paper goes beyond reporting benchmark scores by testing targeted changes to viewpoint, rendering, temporal sampling, and semantic annotation. The use of control subcategories and replication on InternVL2 makes these intervention results practically informative, even if the causal interpretation is currently too strong.
Critical weaknesses
  1. The paper does not validate reliable evaluation of actual generated motion. FineMo pairs clean ground-truth mocap with counterfactual descriptions that alter one semantic dimension at a time. Actual T2M outputs can contain partial matches, physical artifacts, ambiguity, and multiple simultaneous errors. The paper does not evaluate agreement with independent human judgments on outputs from T2M generators. Consequently, the experiments support controlled semantic discrimination but not yet the deployment claim emphasized in the title and abstract.
  2. The quantitative evaluation and root-cause interpretation contain important confounds. Each group has one positive and two negative pairs. Unless the judgment set is rebalanced, an always-“No” classifier obtains 66.7% accuracy, rather than the stated 50% baseline. Moreover, the proposed interventions inject highly informative signals computed from 3D joints: depth maps restore privileged geometry, body-part legends expose topology, and task-specific keyframe algorithms partially compute repetition, displacement, or speed before VLM inference. These gains show that additional signals help, but they do not uniquely demonstrate the proposed internal causes. Direct geometric baselines and statistical uncertainty estimates are also missing.
Questions for authors
  1. Clarification: Is judgment accuracy computed over all positive, hard-negative, and easy-negative pairs without class rebalancing? If so, please report the 66.7% majority baseline, balanced accuracy, per-class recall, and macro-F1. Do the conclusions from Table 2 remain unchanged?
  2. Evidence that could materially affect the assessment: Can the authors evaluate the baseline and adaptive configurations on actual outputs from representative T2M generators, using independent human alignment and localized-error annotations?
  3. Clarification: How do direct geometric algorithms using the same 3D joint data perform on direction, repetition count, motion amplitude, and speed? Does the VLM provide additional value beyond this preprocessing?
  4. Clarification: Which task and sample set are used in Table 3? Does the adaptive framework always use predicted routing rather than construction-derived labels? Please also report the number of independent groups per subcategory and group-level confidence intervals.
3 comments

We thank the reviewer for the detailed and technically precise review. Due to the length limit, our response is split into three comments.

Weakness 1 & Q2: Validation on Actual T2M Outputs

We agree that the original experiments support controlled semantic discrimination rather than deployment. Following Q2, we evaluated the baseline and adaptive configurations on actual T2M outputs with human alignment and localized-error annotations.

Data and annotation. We collected 200 text-motion pairs generated by MDM, MLD, and MoMask from HumanML3D test prompts, sampled so that each of the six dimensions is involved in a comparable number of prompts. Three annotators independently provided: (1) a 1–5 Likert alignment score (Krippendorff's α, ordinal = 0.74); (2) localized errors, i.e., text phrases not realized by the motion and their dimensions (dimension-level Fleiss' κ = 0.68); (3) a flag for visible physical artifacts (foot sliding, jitter, penetration). By majority vote, 54 samples have no error, 86 a single-dimension error, and 60 errors in two or more dimensions; 47 contain physical artifacts.

Protocol. All configurations use Qwen2.5-VL. Paper configurations score alignment by the probability of "Yes" under the judgment prompt; VeMo-Repro re-implements VeMo from its description (skinned mesh, 6 views, likelihood scoring, entropy-based view selection), as its code is not public. Within each base, configurations differ only in the input given to the VLM. Routing uses the diagnosis module of Appendix D.3 and depends only on the text, here the user prompt. The error-correction prompt requires a target dimension, which is unknown for generated motions; we identify the dimensions involved in each text with the LLM decomposition of Appendix A.1, run error correction on each, and take the dimensions reported as erroneous as the localized prediction. For prompts involving several failure-prone semantics, we add a multi-label routing variant that activates only the diagnosed interventions; its prompt was fixed before evaluation on these samples. Keyframes use the index-free, fixed-length setting described under Q3.

ConfigurationSpearman ρ [95% CI]Kendall τLoc. PLoc. RLoc. F1
Paper baseline (skeleton, 1 view)0.39 [0.27, 0.50]0.290.360.290.32
Paper adaptive0.50 [0.39, 0.60]0.380.440.370.40
VeMo-Repro0.48 [0.37, 0.58]0.370.430.350.39
VeMo-Repro + Blanket0.51 [0.40, 0.61]0.390.440.370.40
VeMo-Repro + Adaptive (single-label)0.57 [0.47, 0.66]0.450.500.440.47
VeMo-Repro + Adaptive (multi-label)0.60 [0.51, 0.68]0.470.510.480.49

In a paired bootstrap over samples (10,000 resamples), the adaptive framework improves ρ over its base by +0.11 [0.05, 0.17] (paper) and +0.09 [0.04, 0.14] (VeMo-Repro); multi-label over single-label routing gives +0.03 [−0.01, 0.07].

Stratified by the properties raised by the reviewer (localization recall):

SubsetnVeMo-Repro+ Adaptive (single)+ Adaptive (multi)
Single-dimension error860.410.520.53
Multiple simultaneous errors600.300.370.44
With physical artifacts470.310.380.41
Without physical artifacts1530.370.460.50

Multiple simultaneous errors. Single-label routing yields a smaller gain on multi-error samples (+0.07) than on single-error samples (+0.11), as it activates one intervention per sample. Multi-label routing raises recall on multi-error samples to 0.44 (+0.07 over single-label, [0.01, 0.13]) and is on par with single-label routing on single-error samples.

Physical artifacts. Gains are slightly smaller on samples with artifacts (+0.07 vs. +0.09, n = 47). Among temporal-resolution failures in this subset, we find spurious keyframes caused by jitter; we will report this as a limitation.

Partial matches and ambiguity. The Likert scale records graded partial matches. On the 131 samples where all annotators' scores lie within one point, ρ is 0.53 / 0.62 / 0.65 for VeMo-Repro / single-label / multi-label, in the same order.

We will add this experiment to the main paper and revise the abstract and conclusion to scope the deployment claims to what has been validated: the generators tested, the single- and multi-label routing settings, and the effect of physical artifacts.

Weakness 2 (class balance) & Q1

Yes. Judgment accuracy was computed over all 4,800 pairs without rebalancing (positive:negative = 1:2), so the majority-class baseline is 66.7%. The stated 50% corresponds to uniform random guessing and is not an appropriate chance level. We will correct this and adopt balanced accuracy (chance = 50%) as the primary judgment metric. Six-model averages (metrics computed per model):

DimensionRaw Acc.Balanced Acc.Pos. RecallNeg. RecallMacro-F1
Action Type767988700.75
Temporal Structure707485630.69
Body Part677184590.67
Spatial Direction576382450.57
Motion Intensity515880360.50
Motion Frequency495779340.49
Baseline66.7 (all "No")50——0.40 (all "No")

Under balanced accuracy and macro-F1, the dimension ordering in Table 2 is unchanged. Positive recall is 79–88% across dimensions; the differences come mainly from negative recall: on Spatial Direction, Motion Intensity, and Motion Frequency, models accept 55–66% of mismatched descriptions.

Error correction has the same imbalance: answering "no error" for every pair yields 33.3%. We will report "no error" accuracy on positives and localization accuracy on negatives separately; on Motion Intensity and Motion Frequency these are 31% / 12% and 30% / 6%, and the dimension ordering is unchanged under both.

The same consideration applies to ranking. Because labels are rotated cyclically, a model that always outputs the same label order is correct under exactly one rotation, i.e., 33.3% rather than 17%; across the six models, 4–21% of groups receive an identical order under all three rotations. We add a group-level metric that counts a group as correct only if all three rotations are ranked correctly (chance: 0.5% for random ordering, 0% for any fixed order). Six-model averages are 48 / 36 / 31 / 14 / 9 / 7 for AT / TS / BP / SD / MI / MF, with the same ordering. We will revise the chance-level statements in Section 4 accordingly. Per-model results will be in the appendix.

Weakness 2 (injected signals) & Q3

To clarify the information source: all interventions are computed from the 3D motion being evaluated, which is available at evaluation time; in the Weakness 1 experiment, from the generated motions. The question is then whether the VLM adds value beyond direct geometric computation, which we test below. The body-part legend names segments but carries no motion information; the model still has to judge what each segment does.

We agree that the gains show that additional signals help, but do not by themselves establish the internal mechanism. In the paper, "model-stage" means that after rendering saturates (6 views, 4× FPS), the required information is already present in the input but not used by the model, and accuracy improves once it is provided explicitly. We will rephrase these claims as intervention-based attribution evidence rather than proof of root causes.

Keyframe leakage. In the original setting, the number of keyframes correlates with the repetition count, and frame indices are written into the prompt. We add two controls without indices: (a) uniform sampling with the same number of frames as the keyframes; (b) keyframes padded with uniform frames to 16 frames, so frame count no longer reflects repetition (Qwen2.5-VL, ranking, %):

ConfigurationMFMISpeed
Uniform, 4× FPS363842
(a) Uniform, same count as keyframes293134
Keyframes + indices (paper)485050
(b) Keyframes, fixed 16 frames444747

Control (a) has the same frame count, and thus the same count signal, as the keyframes, yet reaches only 29–34%; frame count alone does not account for the gain. Removing indices and fixing the frame count (b) lowers accuracy by 3–4 points. The revision adopts setting (b) for all keyframe results.

Q3 (continued): Geometric Baselines

An LLM parses the target attribute from the text (direction, count, intensity level, speed level), and rules compute it from 3D joints: displacement direction of the relevant joint in the body-facing frame, cycles of velocity reversal of the primary joint, range of motion mapped to intensity levels, and root velocity mapped to speed levels. Thresholds are calibrated on 20% of FineMo groups; all rows below are reported on the remaining 80% (ranking, %):

MethodSpatial Dir.Front/BackMFMISpeed
Qwen2.5-VL, original input3323222527
Qwen2.5-VL + interventions5249444747
Geometric8486765871

On FineMo, geometry outperforms the VLM on all of these attributes and is the stronger baseline for directly measurable attributes; we will add it to the paper.

On the actual T2M outputs from Weakness 1, for texts involving these attributes (n = 112), attribute-level localization (P / R):

AttributeGeometricVLM + Adaptive (single-label)
Direction0.63 / 0.610.55 / 0.52
Count0.37 / 0.530.49 / 0.46
Intensity0.31 / 0.290.42 / 0.40
Speed0.36 / 0.440.45 / 0.43

Geometry remains more accurate on direction. On count, its recall is higher but precision lower, as jitter produces extra velocity reversals and inflates counts. On intensity, where phrases such as "slightly" must be mapped to fixed thresholds, both metrics are lower; on speed, recall is comparable and precision lower. In 27 cases, annotators mark an error on the checked attribute that the geometric check does not detect; VLM + Adaptive identifies 15 of them. In 58 cases where both geometry and annotators find no error on the attribute, VLM + Adaptive reports an error in 9. Using geometric checks for direction and count, and VLM + Adaptive for the rest, raises Spearman ρ on all 200 samples from 0.57 to 0.61 (paired Δ = +0.04 [0.01, 0.07]). We will report the geometric baselines and include this hybrid configuration in the adaptive framework.

Q4: Setting of Table 3, Routing, and Confidence Intervals

Task and sample set. As in Section 5, Table 3 uses the ranking task with Qwen2.5-VL on all 1,600 FineMo groups, each evaluated under three label rotations, with accuracy averaged over rotations. We will state this in Section 6.2.

Routing. The adaptive framework uses predicted routing from the diagnosis module; the diagnosis input is each group's positive description (Appendix D.3). Since the three texts in a group differ only on the target dimension, we also ran diagnosis on a randomly chosen candidate text: 97% of groups receive the same routing, and overall accuracy is unchanged.

In the original Table 3, however, the keyframe strategy within the temporal-resolution class followed the construction-derived subcategory. We correct this: when the diagnosis is temporal-resolution, the module also outputs the cue type (count, intensity, or speed) and selects the corresponding strategy in Appendix C.3 (sub-type accuracy 96%); for texts without a temporal cue, the union of keyframes from the three strategies is used. This applies to every configuration containing the temporal intervention, with keyframe setting (b) from Q3.

Independent groups per subcategory: depth-related 238 (front/back 150, depth-dependent actions 88); topology-related 208 (lateral 106, chain-adjacent 102); temporal-resolution 512 (MF 250, MI 224, speed 38); non-failure-mode 642.

Re-run under the corrected setting, with 95% confidence intervals from group-level bootstrap (10,000 resamples):

ConfigurationDepthTopologyTemporalNon-FMOverall
Baseline2540256041 [39, 43]
Temporal-Aware2641466249 [47, 51]
Blanket4553345547 [45, 49]
Adaptive (predicted routing)53 [47, 59]61 [54, 68]45 [41, 49]61 [57, 65]55 [53, 57]
Adaptive (construction labels)5361466055 [53, 57]

The Depth-Aware and Topology-Aware rows do not involve keyframes and are unchanged. Overall accuracy of the adaptive framework changes from 57% to 55%, mainly in the temporal category (50% to 45%). Predicted routing and construction labels give the same overall accuracy. In a group-level paired bootstrap, the gain of Adaptive over Blanket is +8 [6, 10], and over the best single intervention +6 [4, 8]. Group-level confidence intervals will also be added to Table 2 and Tables 9–14.

Reviewer_deCuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FineMo to evaluate whether VLMs can detect fine-grained motion-text mismatches. It tests six models across six semantic dimensions and identifies difficulties with depth, body parts, and temporal details. The authors then propose changes to rendering and frame selection, with an adaptive framework that selects an intervention based on the text.

Critical strengths
  • The benchmark provides a useful way to study specific motion-text mismatches by changing one attribute at a time. The three tasks and human validation help assess whether models can identify these errors.
  • The experiments show which input changes help different types of errors. Selecting the relevant intervention works better than applying all interventions together on FineMo.
Critical weaknesses
  1. My main concern is that the proposed improvements are only evaluated on FineMo. It is unclear whether adding them to an existing LLM/VLM-based T2M evaluator would correct its evaluation errors or improve agreement with humans. The results show that the interventions help on the authors’ benchmark, but their practical benefit for T2M evaluation remains unverified. This weakens the paper’s claim of reliable deployment.
  2. The contribution beyond existing T2M evaluation methods needs to be clearer. VeMo [1] already considers view selection, T2MBench [2] uses SMPL rendering and body-part colors, and NextMotionQA [3] studies fine-grained VLM judgment. The depth maps, keyframe selection, and routing may be useful additions, but the comparisons mainly use a single-view skeleton baseline. It is difficult to tell how much they help an evaluator that already uses richer inputs.

[1] Yuwen Ji, Donglin Wang, and Yue Zhang. Zero-Shot Text-to-Motion Evaluation using Video Language Models. ICML, 2026.
[2] Bin Yang, Rong Ou, Weisheng Xu, Jiaqi Xiong, Xintao Li, Taowen Wang, Luyu Zhu, Xu Jiang, Jing Tan, and Renjing Xu. T2MBench: A Benchmark for Out-of-Distribution Text-to-Motion Generation. AACL-IJCNLP, 2026.
[3] Yong Cao, Chuqiao Li, Xianghui Xie, Gerard Pons-Moll, and Andreas Geiger. NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models. arXiv preprint, 2026.

Questions for authors
  1. Can the proposed interventions improve an existing LLM/VLM-based T2M evaluator, such as VeMo [1]? A comparison of the same evaluator before and after adding the interventions, using human judgments as the reference, would help establish their practical benefit. If such results are unavailable, what evidence supports transferring the improvements observed on FineMo to this setting?
  2. Which components provide new contributions beyond the rendering and view-selection choices already used in existing T2M evaluation methods? Do the proposed additions or routing still help when applied on top of those configurations?

Minor suggestions

  • Introduction: The four-part study overview and the contribution list are repetitive.
  • Figs. 1–4: The internal text, axis labels, and legends are too small. Please enlarge them and simplify crowded layouts. Vector export would help, especially for Fig. 1.
  • Tabs. 1–3: The font is much smaller than the body text. Please increase it for readability.
  • It would be interesting to test newer models with stronger 3D perception. Testing newly released models would not change my judgment either.

These are minor suggestions and do not require a rebuttal response.

3 comments

We thank the reviewer for the careful reading and constructive suggestions. Both critical weaknesses point to a core question: does FineMo's analysis lead to practical improvements for real T2M evaluation? We agree this is the main gap and address it with two new experiments.

Weakness 1 & Q1: Practical Benefit for Real T2M Evaluation

New Experiment. We evaluate whether the adaptive framework improves a VeMo-style evaluator's agreement with human judgments. Since VeMo's code is not publicly available, we re-implement its core pipeline: SMPL skinned-mesh rendering, 6-azimuth multi-view rendering, Qwen2.5-VL normalized-likelihood scoring, and entropy-driven view selection. This baseline already incorporates the skinned mesh and multi-view rendering that the reviewer highlights.

We collect 200 text-motion pairs from three T2M models (MDM, MLD, MoMask) on HumanML3D test prompts, balanced across six semantic dimensions. Three annotators rate text-motion alignment on a 1–5 Likert scale (inter-annotator κ=0.74). Three conditions:

  • (A) VeMo-Repro: skinned mesh + 6-view + entropy selection + uniform 15 FPS
  • (B) + Blanket: all samples receive depth maps + keyframe extraction
  • (C) + Adaptive (Ours): per-sample failure-mode diagnosis → matched intervention only

Following VeMo's evaluation protocol, we report Spearman ρ and Kendall τ against mean human scores:

ConfigurationSpearman ρKendall τ
(A) VeMo-Repro0.480.37
(B) + Blanket0.510.39
(C) + Adaptive (Ours)0.570.45

The adaptive framework improves ρ by +0.09 and τ by +0.08 on top of this strong baseline. Per-category results:

Text Category(A) ρ(C) ρΔρ
Depth-related0.330.52+0.19
Topology-related0.490.56+0.07
Temporal-resolution0.310.50+0.19
Non-failure-mode0.630.64+0.01

Gains concentrate on depth-related and temporal-resolution texts, consistent with the FineMo analysis (Sections 5.1, 5.3). Topology-related gains are smaller because the baseline's skinned mesh already provides rich body-surface cues. Non-failure-mode texts show negligible change, indicating the framework introduces no noise when intervention is unnecessary. Blanket (ρ=0.51) improves far less than Adaptive (ρ=0.57), showing that applying all interventions indiscriminately does not fully realize each intervention's potential.

Weakness 2 & Q2: Incremental Contribution over Existing Methods

We agree that the original ablations start from a skeleton-line baseline, while VeMo uses skinned mesh + multi-view, T2MBench uses SMPL + body-part colors, and NextMotionQA uses SMPL-H + body-region coloring. To examine incremental value on top of these, we conduct the following experiment.

Experiment. Strong baseline: SMPL skinned mesh + body-part color annotation + 6-view rendering (union of techniques from the three cited works). Evaluated on FineMo ranking with Qwen2.5-VL.

ConfigurationDepthTopologyTemporalNon-FMOverall
Paper baseline (skeleton, 1-view, 15FPS)2540256041
Strong Baseline (mesh+colors+6-view)4360276350
Strong Baseline + Blanket4856375850
Strong Baseline + Adaptive (Ours)5261446356

The strong baseline raises overall from 41% to 50%, with topology from 40% to 60% and depth from 25% to 43%, confirming the reviewer's observation that existing rendering techniques account for a substantial portion of the gains.

On top of this, Blanket applies all interventions to every sample but does not improve over the strong baseline (50% vs. 50%): gains on target categories are offset by a drop on non-FM samples (63%→58%). The adaptive framework raises overall from 50% to 56% (+6pp) by activating only the matched intervention per sample—depth-related samples receive depth maps (43%→52%), temporal-related samples receive keyframe extraction (27%→44%), and samples needing no intervention remain unaffected (63%).

These results show that selectively matching interventions to samples is more effective than applying a fixed configuration to all. This is how our framework complements existing methods—VeMo, T2MBench, and NextMotionQA provide effective rendering and scoring, and our diagnosis-and-routing layer helps these components work more precisely.

Minor Suggestions

We will address all presentation suggestions in the revision.

Thank you for the rebuttal and the additional experiments. I appreciate the direction of this work and the evidence that the proposed interventions improve agreement with human ratings on generated motions. The gains are encouraging, but the current results fall short of supporting the claim of reliable deployment. Therefore I am keeping my original rating of weak rejection.

Thank you for the response and for acknowledging the new experiments. We agree that the current evidence supports improved agreement with human judgments rather than reliable deployment. We will revise the claims accordingly and add the following results to make the strength of the evidence clearer.

Revised claims. "Reliable Deployment" in the title will become "More Reliable Deployment". In the abstract and conclusion, "reliable deployment" will be replaced by a specific statement: on the generators and VLMs tested, the adaptive framework improves the evaluator's agreement with human judgments, and a gap to human-level agreement remains.

Human agreement as a reference. The Spearman ρ between each annotator and the mean of the other two averages 0.71 [0.64, 0.77], which we use as a reference upper bound for an evaluator. VeMo-Repro reaches 0.48 and 0.57 with the adaptive framework; both remain below this bound.

Statistical significance. In a sample-level paired bootstrap (10,000 resamples), the gain of the adaptive framework over VeMo-Repro is Δρ = +0.09 [0.04, 0.14], and over Blanket +0.06 [0.01, 0.11].

Consistency across generators and VLMs. Computed per generator, the gain is positive for each: MDM +0.11, MLD +0.08, MoMask +0.07 (about 67 samples each, so the intervals are wide). Under the paper's unified input, the adaptive framework also improves agreement with human ratings for other VLMs: GPT-5.4 from 0.44 to 0.52, and InternVL2 from 0.29 to 0.37.

Updated setting. Following another reviewer's comment on information leakage in keyframes, we removed frame indices and fixed the number of frames. Under this setting, ρ of the adaptive framework in the generated-motion experiment remains 0.57; in the FineMo strong-baseline experiment, overall accuracy of Adaptive changes from 56% to 55%, and the other conclusions are unchanged.

Our responses to the other reviewers also report error localization against human-annotated error dimensions, and results stratified by samples with multiple simultaneous errors and with physical artifacts. If any specific aspect of the deployment claim still seems insufficiently supported, we would be glad to provide further evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.