- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces STRIDE, a diagnostic benchmark for evaluating fine-grained structural grounding in motion-language evaluators. The benchmark contains 5,869 motion-caption triplets constructed from HumanML3D and SnapMoGen, covering three types of perturbations: Temporal Order, Mirror Reflection, and Action Identity.
To mitigate language-prior bias, the authors apply likelihood-based filtering and introduce unrelated-motion controls. They further propose two evaluation measures, normalized grounding gain (Δ) and Grounded Perturbation Recognition (GPR), to quantify whether matched motion information improves structural discrimination.
Experiments on six released motion-language evaluators reveal substantial performance gaps between coarse content matching and fine-grained structural understanding, particularly for left-right distinctions. The authors also demonstrate that standard motion-generation metrics are relatively insensitive to certain structural errors. Finally, they investigate structural hard-negative training as a simple intervention to improve grounding performance.
Critical strengths
- The paper is well-written and addresses an important limitation of motion-language evaluation: conventional metrics may overlook fine-grained spatial and temporal relationships.
- The proposed benchmark covers complementary perturbations, short and long descriptions, multiple evaluators, and unrelated-motion controls.
- Experiments expose weaknesses in existing evaluators, especially for mirror reflections and structural inconsistencies, while showing that hard-negative supervision can improve discrimination without architectural changes.
Critical weaknesses
- The methodological novelty is limited, as STRIDE largely adapts established compositional perturbation and hard-negative learning techniques to motion-language evaluation. Its differences from prior benchmarks and training strategies should be clarified.
- Benchmark reliability and coverage require stronger validation due to imbalanced perturbation categories, residual text-only cues, and insufficient reporting of the human validation protocol.
- The proposed grounding metrics require further validation, as unrelated-motion sampling may introduce accidental semantic similarities, while Δ and GPR may be sensitive to baseline estimation.
- The practical significance remains unclear because experiments mainly use controlled ground-truth perturbations rather than real generator outputs or human judgments.
- The effectiveness and generalization of structural hard-negative training are insufficiently demonstrated due to inconsistent gains, missing multi-seed results and training controls, and limited evaluation on unseen perturbations.
Missing. some related work:
The related-work discussion could be strengthened by positioning STRIDE more explicitly against prior diagnostic benchmarks such as ViLMA (ICLR 2024), as well as recent text-to-motion evaluation frameworks such as VeMo (ICML 2026). Although these works focus on different modalities or evaluation settings, they share related motivations concerning fine-grained grounding and evaluation reliability. A clearer discussion would help establish the distinctive contribution of STRIDE.
Questions for authors
- Could the authors provide detailed human-validation statistics and explain why Action Identity contains substantially fewer samples?
- How sensitive are Δ and GPR to unrelated-motion selection, and would multiple samples or bidirectional consistency analysis change the conclusions?
- Can STRIDE better evaluate direction-sensitive captions and actual generator outputs, particularly in agreement with human judgments?
- Could the authors compare structural hard-negative training with standard contrastive fine-tuning under the same budget and provide multi-seed and ablation results?
- How well do the improvements generalize to unseen structural variations, and can cross-dataset confounding factors be disentangled?
8 comments
Response to reviewer aCwJ
W2: Benchmark reliability and coverage require stronger validation due to imbalanced perturbation categories, residual text-only cues, and insufficient reporting of the human validation protocol.
Q1: Could the authors provide detailed human-validation statistics and explain why Action Identity contains substantially fewer samples?
A3: Thank you for these reliability questions. No audited items required revision or exclusion, and equal-size testing and stricter filtering preserve the diagnosis. AI is smaller because valid, balanced replacements are more restrictive. Revised Appendix B and Appendix F.1, Tables 11–15, address ambiguity, sampling uncertainty, and text cues separately. With motion visible, the same five annotators unanimously confirmed caption quality and an unambiguous match. No item required revision or exclusion.
Category sizes. AI requires visually distinct, mutually exclusive predicate replacements that preserve other content. Deduplication, likelihood filtering, and balancing replacement directions further reduce eligible pairs.
Equal-size test. Matching category sizes preserves the diagnosis. We evaluated all 11 settings with 112 items per axis on HumanML3D and 204 on SnapMoGen, reporting mean Δ over ten draws.
Table 11: Equal-size evaluation — HumanML3D (112 per axis).
| Evaluator | TO Δ | MR Δ | AI Δ | RC Δ |
|---|---|---|---|---|
| H3D eval. | 0.422 | 0.058 | 0.744 | 0.873 |
| TMR | 0.404 | 0.427 | 0.777 | 0.909 |
| CLaM | 0.505 | 0.453 | 0.860 | 0.958 |
| MoBERT | 0.430 | 0.219 | 0.705 | 0.884 |
| ChronRet | 0.389 | 0.393 | 0.852 | 0.954 |
Table 12: Equal-size evaluation — SnapMoGen (204 per axis).
| Evaluator | TO Δ | MR Δ | AI Δ | RC Δ |
|---|---|---|---|---|
| H3D eval. | 0.063 | 0.002 | 0.125 | 0.468 |
| TMR | 0.063 | 0.229 | 0.298 | 0.586 |
| CLaM | 0.124 | 0.199 | 0.279 | 0.707 |
| MoBERT | 0.083 | 0.119 | 0.163 | 0.440 |
| ChronRet | 0.135 | 0.143 | 0.207 | 0.627 |
| Snap eval. | 0.919 | 0.466 | 0.668 | 0.989 |
Every structural-axis Δ remains below RC in all 11 settings and every draw. SnapMoGen retains its TO–MR gap (0.919 versus 0.466). Equal category sizes thus preserve the main diagnosis.
Estimation uncertainty. Despite AI’s lower precision, the intervals and equal-size results support the statistical reliability of the main diagnosis. The 95% bootstrap CIs below account for related items sharing a source motion.
Table 13: HumanML3D Δ — 95% confidence intervals.
| Evaluator | TO Δ | MR Δ | AI Δ |
|---|---|---|---|
| H3D eval. | [0.322, 0.455] | [-0.052, 0.074] | [0.605, 0.862] |
| TMR | [0.336, 0.486] | [0.367, 0.487] | [0.625, 0.902] |
| CLaM | [0.432, 0.554] | [0.381, 0.502] | [0.755, 0.946] |
| MoBERT | [0.344, 0.491] | [0.109, 0.243] | [0.550, 0.834] |
| ChronRet | [0.325, 0.467] | [0.303, 0.421] | [0.725, 0.945] |
Table 14: SnapMoGen Δ — 95% confidence intervals.
| Evaluator | TO Δ | MR Δ | AI Δ |
|---|---|---|---|
| H3D eval. | [-0.009, 0.145] | [-0.055, 0.043] | [0.056, 0.196] |
| TMR | [0.016, 0.130] | [0.140, 0.265] | [0.155, 0.431] |
| CLaM | [0.046, 0.192] | [0.127, 0.258] | [0.127, 0.415] |
| MoBERT | [0.041, 0.160] | [0.037, 0.143] | [0.056, 0.265] |
| ChronRet | [0.053, 0.179] | [0.074, 0.214] | [0.056, 0.350] |
| Snap eval. | [0.900, 0.952] | [0.378, 0.504] | [0.552, 0.774] |
Lower precision does not imply invalid items or systematically lower scores; equal-size tests retain the structural-versus-RC pattern.
Residual text cues. Stricter filtering preserves the diagnosis, supporting robustness to the tested residual text bias. We tightened the likelihood-difference threshold from 0.15 to 0.05 nats/token, using language scores alone for selection. The table reports native-dataset Δ and retained counts.
Table 15: Stricter language filtering.
| Evaluator | Axis | 0.15 | 0.05 |
|---|---|---|---|
| H3D eval. | TO | 0.390 (953) | 0.381 (327) |
| H3D eval. | MR | 0.013 (1000) | -0.023 (649) |
| H3D eval. | AI | 0.744 (112) | 0.767 (35) |
| Snap eval. | TO | 0.928 (1000) | 0.949 (339) |
| Snap eval. | MR | 0.444 (1000) | 0.440 (892) |
| Snap eval. | AI | 0.668 (204) | 0.665 (133) |
The structural weakness persists after stricter filtering, supporting our conclusion that evaluators ground fine-grained structure less effectively than overall action content.
Response to Reviewer aCwJ
W3: The proposed grounding metrics require further validation, as unrelated-motion sampling may introduce accidental semantic similarities, while Δ and GPR may be sensitive to baseline estimation.
Q2: How sensitive are Δ and GPR to unrelated-motion selection?
A4: Thank you for highlighting these baseline concerns. Neither unrelated-motion resampling nor semantic exclusion changes the main diagnosis. Revised Table 3 and Appendix F.2, Table 16, report these checks. Resampling tests baseline variation; exclusion tests whether accidentally shared action semantics drive the result. We resampled unrelated motions in ten rounds across all evaluators; the table reports mean ± SD. Controls come from the same dataset and axis, excluding the matched source and its mirrors.
Table 3: Control-motion resampling on HumanML3D and SnapMoGen.
| Evaluator | TO Δ | MR Δ | AI Δ | RC Δ | Macro GPR (%) |
|---|---|---|---|---|---|
| HumanML3D | |||||
| HumanML3D evaluator | 0.390 ± 0.015 | 0.013 ± 0.024 | 0.744 ± 0.015 | 0.880 ± 0.005 | 66.8 ± 1.4 |
| TMR | 0.412 ± 0.016 | 0.430 ± 0.016 | 0.777 ± 0.020 | 0.911 ± 0.002 | 72.0 ± 1.5 |
| CLaM | 0.495 ± 0.012 | 0.443 ± 0.019 | 0.860 ± 0.007 | 0.955 ± 0.002 | 75.2 ± 1.3 |
| MoBERT | 0.423 ± 0.020 | 0.179 ± 0.020 | 0.705 ± 0.019 | 0.877 ± 0.006 | 70.2 ± 0.9 |
| ChronRet | 0.403 ± 0.023 | 0.364 ± 0.019 | 0.852 ± 0.015 | 0.939 ± 0.003 | 72.8 ± 0.6 |
| SnapMoGen | |||||
| HumanML3D evaluator | 0.069 ± 0.030 | -0.004 ± 0.024 | 0.125 ± 0.040 | 0.494 ± 0.012 | 52.1 ± 0.8 |
| TMR | 0.074 ± 0.020 | 0.204 ± 0.022 | 0.298 ± 0.037 | 0.572 ± 0.017 | 57.8 ± 1.1 |
| CLaM | 0.120 ± 0.017 | 0.194 ± 0.016 | 0.279 ± 0.025 | 0.687 ± 0.011 | 57.8 ± 0.7 |
| MoBERT | 0.101 ± 0.025 | 0.089 ± 0.021 | 0.163 ± 0.026 | 0.461 ± 0.010 | 55.4 ± 0.4 |
| ChronRet | 0.119 ± 0.017 | 0.146 ± 0.024 | 0.207 ± 0.051 | 0.635 ± 0.013 | 57.6 ± 1.4 |
| SnapMoGen evaluator | 0.928 ± 0.002 | 0.444 ± 0.014 | 0.668 ± 0.020 | 0.995 ± 0.000 | 74.3 ± 0.7 |
All ten resampling rounds preserve structural Δ below RC in every setting. SnapMoGen’s mean TO–MR gap remains 0.928 vs. 0.444; Macro GPR SDs range from 0.4 to 1.5 percentage points. The diagnosis is stable under unrelated-motion resampling.
Semantic exclusion. Different motion IDs can still share action content. We therefore repeated evaluation after excluding controls whose captions were semantically similar to the target pair. Results average all 11 settings. Differences are With − Without; GPR differences are percentage points.
Table 16: Semantic exclusion.
| Semantic exclusion | TO Δ | MR Δ | AI Δ | RC Δ | Macro GPR (%) |
|---|---|---|---|---|---|
| With | 0.32345 | 0.22294 | 0.51878 | 0.76588 | 64.9 |
| Without | 0.32169 | 0.22097 | 0.51445 | 0.76503 | 64.7 |
| Difference | +0.00176 | +0.00197 | +0.00433 | +0.00085 | +0.2 |
Average shifts are small; the largest absolute mean-Δ change for any setting/axis is 0.040. Every tested assignment retains structural Δ below RC. Accidental similarity under this criterion therefore does not change the diagnostic conclusion.
Response to reviewer aCwJ
Thank you for the detailed review. We have clarified STRIDE’s contribution and expanded validation of the benchmark, controls, and training intervention, including held-out variations and generated motions with human judgments. We address the related comments together below.
A1: Thank you for suggesting clearer positioning. STRIDE’s distinctive contribution is a controlled diagnosis of whether motion supplies structural discrimination, beyond correct caption selection. Revised Section 5 clarifies its relationship to ViLMA and VeMo. Perturbation accuracy alone cannot separate grounding from caption preference.
Likelihood balancing reduces dataset-level language bias. Scoring the same caption pair with matched and unrelated motions then measures each evaluator’s residual preference and the discrimination added by motion. Comparing three structural axes against RC separates structural grounding from content matching; motion-side perturbations expose errors missed by standard generation metrics.
The revised related work positions STRIDE alongside ViLMA and VeMo:
Related-work comparison
The hard-negative intervention supports the diagnostic findings by testing whether targeted supervision improves the identified weaknesses.
A2: Thank you for asking about practical relevance. Structural supervision also improves agreement with human judgments on generated motions. Revised Table 6 and Appendix G report the results and protocol. Humans label actual outputs because generation prompts may not be followed. We generated 90 MoMask motions (30 per axis, with 15 contrasting prompt pairs per axis).
Five annotators provided majority labels. Nine “neither” and one “both” item were excluded, leaving 80 exclusive A/B items (22 TO, 28 MR, 30 AI). The table reports human agreement (%) for four released evaluators and three-seed HumanML3D/TMR fine-tuning; Macro weights axes equally.
Table 6: Agreement with human judgments on generated motions.
HumanML3D MR agreement rises from 48.8% to 77.4% and TMR TO from 63.6% to 77.3% over Vanilla FT. Thus, structural supervision can improve decisions about actual generated motion, not merely recovery of the intended prompt.