acceptodds
Desk rejected at ICLR 2027

Can Motion-Language Models Ground Structure? STRIDE for Evaluating the Evaluators

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Motion-language models are typically scored by motion-language evaluators, but how well these evaluators ground language structure remains unclear. Here, we introduce the Structure grounding via Temporal-order, Reflection, and Identity Diagnostic Evaluation (STRIDE) benchmark to systematically evaluate the ability of evaluators to track temporal order, mirror reflection, and action identity. STRIDE comprises 5,869 triples, each consisting of a motion, its original caption, and a perturbed caption, spanning both short and long descriptions. We likelihood-balance caption pairs to reduce text-only bias and estimate each evaluator’s caption preference under unrelated motions to measure the discrimination gain from matched motions relative to this baseline. Our experiments reveal weak structural grounding and severe deficits in mirror sensitivity among the audited evaluators, which commonly used evaluation protocols fail to expose. To understand why these limitations go undetected in standard tests, we examine the evaluators more closely. We find that text-only priors alone can solve naive perturbation tests on existing datasets, while common retrieval and distributional metrics barely respond to structural corruption introduced by mirroring ground-truth motions. These findings suggest a natural intervention: structural hard negatives. Our experiments show that a simple modification to contrastive learning substantially improves performance on temporal order and mirror reflection. The benchmark and code will be released.

Primary area:
datasets and benchmarks
Keywords:
Motion-Language Models, Structural Grounding, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aCwJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces STRIDE, a diagnostic benchmark for evaluating fine-grained structural grounding in motion-language evaluators. The benchmark contains 5,869 motion-caption triplets constructed from HumanML3D and SnapMoGen, covering three types of perturbations: Temporal Order, Mirror Reflection, and Action Identity.

To mitigate language-prior bias, the authors apply likelihood-based filtering and introduce unrelated-motion controls. They further propose two evaluation measures, normalized grounding gain (Δ) and Grounded Perturbation Recognition (GPR), to quantify whether matched motion information improves structural discrimination.

Experiments on six released motion-language evaluators reveal substantial performance gaps between coarse content matching and fine-grained structural understanding, particularly for left-right distinctions. The authors also demonstrate that standard motion-generation metrics are relatively insensitive to certain structural errors. Finally, they investigate structural hard-negative training as a simple intervention to improve grounding performance.

Critical strengths
  • The paper is well-written and addresses an important limitation of motion-language evaluation: conventional metrics may overlook fine-grained spatial and temporal relationships.
  • The proposed benchmark covers complementary perturbations, short and long descriptions, multiple evaluators, and unrelated-motion controls.
  • Experiments expose weaknesses in existing evaluators, especially for mirror reflections and structural inconsistencies, while showing that hard-negative supervision can improve discrimination without architectural changes.
Critical weaknesses
  • The methodological novelty is limited, as STRIDE largely adapts established compositional perturbation and hard-negative learning techniques to motion-language evaluation. Its differences from prior benchmarks and training strategies should be clarified.
  • Benchmark reliability and coverage require stronger validation due to imbalanced perturbation categories, residual text-only cues, and insufficient reporting of the human validation protocol.
  • The proposed grounding metrics require further validation, as unrelated-motion sampling may introduce accidental semantic similarities, while Δ and GPR may be sensitive to baseline estimation.
  • The practical significance remains unclear because experiments mainly use controlled ground-truth perturbations rather than real generator outputs or human judgments.
  • The effectiveness and generalization of structural hard-negative training are insufficiently demonstrated due to inconsistent gains, missing multi-seed results and training controls, and limited evaluation on unseen perturbations.

Missing. some related work:

The related-work discussion could be strengthened by positioning STRIDE more explicitly against prior diagnostic benchmarks such as ViLMA (ICLR 2024), as well as recent text-to-motion evaluation frameworks such as VeMo (ICML 2026). Although these works focus on different modalities or evaluation settings, they share related motivations concerning fine-grained grounding and evaluation reliability. A clearer discussion would help establish the distinctive contribution of STRIDE.

Questions for authors
  • Could the authors provide detailed human-validation statistics and explain why Action Identity contains substantially fewer samples?
  • How sensitive are Δ and GPR to unrelated-motion selection, and would multiple samples or bidirectional consistency analysis change the conclusions?
  • Can STRIDE better evaluate direction-sensitive captions and actual generator outputs, particularly in agreement with human judgments?
  • Could the authors compare structural hard-negative training with standard contrastive fine-tuning under the same budget and provide multi-seed and ablation results?
  • How well do the improvements generalize to unseen structural variations, and can cross-dataset confounding factors be disentangled?
8 comments
AuthorsOpenReview ↗

Response to reviewer aCwJ

Thank you for the detailed review. We have clarified STRIDE’s contribution and expanded validation of the benchmark, controls, and training intervention, including held-out variations and generated motions with human judgments. We address the related comments together below.

W1: The methodological novelty is limited, as STRIDE largely adapts established compositional perturbation and hard-negative learning techniques to motion-language evaluation. Its differences from prior benchmarks and training strategies should be clarified.

W6: Missing. some related work: The related-work discussion could be strengthened by positioning STRIDE more explicitly against prior diagnostic benchmarks such as ViLMA (ICLR 2024), as well as recent text-to-motion evaluation frameworks such as VeMo (ICML 2026). Although these works focus on different modalities or evaluation settings, they share related motivations concerning fine-grained grounding and evaluation reliability. A clearer discussion would help establish the distinctive contribution of STRIDE.

A1: Thank you for suggesting clearer positioning. STRIDE’s distinctive contribution is a controlled diagnosis of whether motion supplies structural discrimination, beyond correct caption selection. Revised Section 5 clarifies its relationship to ViLMA and VeMo. Perturbation accuracy alone cannot separate grounding from caption preference.

Likelihood balancing reduces dataset-level language bias. Scoring the same caption pair with matched and unrelated motions then measures each evaluator’s residual preference and the discrimination added by motion. Comparing three structural axes against RC separates structural grounding from content matching; motion-side perturbations expose errors missed by standard generation metrics.

The revised related work positions STRIDE alongside ViLMA and VeMo:

Related-work comparison

Prior workConnection to STRIDE and distinction in focus
ViLMAUses counterfactual tests and associated proficiency tests to diagnose linguistic and temporal grounding in video-language models, with automatic and manual validation to control biases. STRIDE combines language-likelihood balancing with an evaluator-specific unrelated-motion baseline, measuring the change in caption discrimination when matched motion replaces unrelated motion.
VeMoUses rendered motions and pretrained video-language models to evaluate generated text-to-motion outputs, with human judgments for meta-evaluation. STRIDE focuses on diagnosing whether an evaluator's fine-grained decisions are supported by motion information after accounting for its residual caption preferences.

The hard-negative intervention supports the diagnostic findings by testing whether targeted supervision improves the identified weaknesses.


W4: The practical significance remains unclear because experiments mainly use controlled ground-truth perturbations rather than real generator outputs or human judgments.

Q3: Can STRIDE better evaluate direction-sensitive captions and actual generator outputs, particularly in agreement with human judgments?

A2: Thank you for asking about practical relevance. Structural supervision also improves agreement with human judgments on generated motions. Revised Table 6 and Appendix G report the results and protocol. Humans label actual outputs because generation prompts may not be followed. We generated 90 MoMask motions (30 per axis, with 15 contrasting prompt pairs per axis).

Five annotators provided majority labels. Nine “neither” and one “both” item were excluded, leaving 80 exclusive A/B items (22 TO, 28 MR, 30 AI). The table reports human agreement (%) for four released evaluators and three-seed HumanML3D/TMR fine-tuning; Macro weights axes equally.

Table 6: Agreement with human judgments on generated motions.

Evaluator / configurationTOMRAIMacro
HumanML3D evaluator50.046.493.363.2
+ Vanilla FT57.6 ± 2.648.8 ± 5.595.6 ± 1.967.3 ± 2.4
+ Neg Suite59.1 ± 4.577.4 ± 2.195.6 ± 1.977.4 ± 1.6
TMR72.7100.0100.090.9
+ Vanilla FT63.6 ± 0.0100.0 ± 0.0100.0 ± 0.087.9 ± 0.0
+ Neg Suite77.3 ± 0.0100.0 ± 0.0100.0 ± 0.092.4 ± 0.0
CLaM45.596.4100.080.6
MoBERT54.582.196.777.8

HumanML3D MR agreement rises from 48.8% to 77.4% and TMR TO from 63.6% to 77.3% over Vanilla FT. Thus, structural supervision can improve decisions about actual generated motion, not merely recovery of the intended prompt.

AuthorsOpenReview ↗

Response to reviewer aCwJ

W2: Benchmark reliability and coverage require stronger validation due to imbalanced perturbation categories, residual text-only cues, and insufficient reporting of the human validation protocol.

Q1: Could the authors provide detailed human-validation statistics and explain why Action Identity contains substantially fewer samples?

A3: Thank you for these reliability questions. No audited items required revision or exclusion, and equal-size testing and stricter filtering preserve the diagnosis. AI is smaller because valid, balanced replacements are more restrictive. Revised Appendix B and Appendix F.1, Tables 11–15, address ambiguity, sampling uncertainty, and text cues separately. With motion visible, the same five annotators unanimously confirmed caption quality and an unambiguous match. No item required revision or exclusion.

Category sizes. AI requires visually distinct, mutually exclusive predicate replacements that preserve other content. Deduplication, likelihood filtering, and balancing replacement directions further reduce eligible pairs.

Equal-size test. Matching category sizes preserves the diagnosis. We evaluated all 11 settings with 112 items per axis on HumanML3D and 204 on SnapMoGen, reporting mean Δ over ten draws.

Table 11: Equal-size evaluation — HumanML3D (112 per axis).

EvaluatorTO ΔMR ΔAI ΔRC Δ
H3D eval.0.4220.0580.7440.873
TMR0.4040.4270.7770.909
CLaM0.5050.4530.8600.958
MoBERT0.4300.2190.7050.884
ChronRet0.3890.3930.8520.954

Table 12: Equal-size evaluation — SnapMoGen (204 per axis).

EvaluatorTO ΔMR ΔAI ΔRC Δ
H3D eval.0.0630.0020.1250.468
TMR0.0630.2290.2980.586
CLaM0.1240.1990.2790.707
MoBERT0.0830.1190.1630.440
ChronRet0.1350.1430.2070.627
Snap eval.0.9190.4660.6680.989

Every structural-axis Δ remains below RC in all 11 settings and every draw. SnapMoGen retains its TO–MR gap (0.919 versus 0.466). Equal category sizes thus preserve the main diagnosis.

Estimation uncertainty. Despite AI’s lower precision, the intervals and equal-size results support the statistical reliability of the main diagnosis. The 95% bootstrap CIs below account for related items sharing a source motion.

Table 13: HumanML3D Δ — 95% confidence intervals.

EvaluatorTO ΔMR ΔAI Δ
H3D eval.[0.322, 0.455][-0.052, 0.074][0.605, 0.862]
TMR[0.336, 0.486][0.367, 0.487][0.625, 0.902]
CLaM[0.432, 0.554][0.381, 0.502][0.755, 0.946]
MoBERT[0.344, 0.491][0.109, 0.243][0.550, 0.834]
ChronRet[0.325, 0.467][0.303, 0.421][0.725, 0.945]

Table 14: SnapMoGen Δ — 95% confidence intervals.

EvaluatorTO ΔMR ΔAI Δ
H3D eval.[-0.009, 0.145][-0.055, 0.043][0.056, 0.196]
TMR[0.016, 0.130][0.140, 0.265][0.155, 0.431]
CLaM[0.046, 0.192][0.127, 0.258][0.127, 0.415]
MoBERT[0.041, 0.160][0.037, 0.143][0.056, 0.265]
ChronRet[0.053, 0.179][0.074, 0.214][0.056, 0.350]
Snap eval.[0.900, 0.952][0.378, 0.504][0.552, 0.774]

Lower precision does not imply invalid items or systematically lower scores; equal-size tests retain the structural-versus-RC pattern.

Residual text cues. Stricter filtering preserves the diagnosis, supporting robustness to the tested residual text bias. We tightened the likelihood-difference threshold from 0.15 to 0.05 nats/token, using language scores alone for selection. The table reports native-dataset Δ and retained counts.

Table 15: Stricter language filtering.

EvaluatorAxis0.150.05
H3D eval.TO0.390 (953)0.381 (327)
H3D eval.MR0.013 (1000)-0.023 (649)
H3D eval.AI0.744 (112)0.767 (35)
Snap eval.TO0.928 (1000)0.949 (339)
Snap eval.MR0.444 (1000)0.440 (892)
Snap eval.AI0.668 (204)0.665 (133)

The structural weakness persists after stricter filtering, supporting our conclusion that evaluators ground fine-grained structure less effectively than overall action content.

AuthorsOpenReview ↗

Response to Reviewer aCwJ

W3: The proposed grounding metrics require further validation, as unrelated-motion sampling may introduce accidental semantic similarities, while Δ and GPR may be sensitive to baseline estimation.

Q2: How sensitive are Δ and GPR to unrelated-motion selection?

A4: Thank you for highlighting these baseline concerns. Neither unrelated-motion resampling nor semantic exclusion changes the main diagnosis. Revised Table 3 and Appendix F.2, Table 16, report these checks. Resampling tests baseline variation; exclusion tests whether accidentally shared action semantics drive the result. We resampled unrelated motions in ten rounds across all evaluators; the table reports mean ± SD. Controls come from the same dataset and axis, excluding the matched source and its mirrors.

Table 3: Control-motion resampling on HumanML3D and SnapMoGen.

EvaluatorTO ΔMR ΔAI ΔRC ΔMacro GPR (%)
HumanML3D
HumanML3D evaluator0.390 ± 0.0150.013 ± 0.0240.744 ± 0.0150.880 ± 0.00566.8 ± 1.4
TMR0.412 ± 0.0160.430 ± 0.0160.777 ± 0.0200.911 ± 0.00272.0 ± 1.5
CLaM0.495 ± 0.0120.443 ± 0.0190.860 ± 0.0070.955 ± 0.00275.2 ± 1.3
MoBERT0.423 ± 0.0200.179 ± 0.0200.705 ± 0.0190.877 ± 0.00670.2 ± 0.9
ChronRet0.403 ± 0.0230.364 ± 0.0190.852 ± 0.0150.939 ± 0.00372.8 ± 0.6
SnapMoGen
HumanML3D evaluator0.069 ± 0.030-0.004 ± 0.0240.125 ± 0.0400.494 ± 0.01252.1 ± 0.8
TMR0.074 ± 0.0200.204 ± 0.0220.298 ± 0.0370.572 ± 0.01757.8 ± 1.1
CLaM0.120 ± 0.0170.194 ± 0.0160.279 ± 0.0250.687 ± 0.01157.8 ± 0.7
MoBERT0.101 ± 0.0250.089 ± 0.0210.163 ± 0.0260.461 ± 0.01055.4 ± 0.4
ChronRet0.119 ± 0.0170.146 ± 0.0240.207 ± 0.0510.635 ± 0.01357.6 ± 1.4
SnapMoGen evaluator0.928 ± 0.0020.444 ± 0.0140.668 ± 0.0200.995 ± 0.00074.3 ± 0.7

All ten resampling rounds preserve structural Δ below RC in every setting. SnapMoGen’s mean TO–MR gap remains 0.928 vs. 0.444; Macro GPR SDs range from 0.4 to 1.5 percentage points. The diagnosis is stable under unrelated-motion resampling.

Semantic exclusion. Different motion IDs can still share action content. We therefore repeated evaluation after excluding controls whose captions were semantically similar to the target pair. Results average all 11 settings. Differences are With − Without; GPR differences are percentage points.

Table 16: Semantic exclusion.

Semantic exclusionTO ΔMR ΔAI ΔRC ΔMacro GPR (%)
With0.323450.222940.518780.7658864.9
Without0.321690.220970.514450.7650364.7
Difference+0.00176+0.00197+0.00433+0.00085+0.2

Average shifts are small; the largest absolute mean-Δ change for any setting/axis is 0.040. Every tested assignment retains structural Δ below RC. Accidental similarity under this criterion therefore does not change the diagnostic conclusion.

Reviewer_bCEUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces STRIDE, a diagnostic benchmark for evaluating fine-grained structural grounding in motion-language evaluators, covering temporal order, mirror reflection, and action identity. It constructs 5,869 motion-caption triplets and incorporates likelihood balancing and unrelated-motion controls to mitigate text-only biases and assess motion-grounded discrimination. Experiments across six released evaluators reveal substantial gaps between content matching and structural understanding, particularly in mirror reflection, while demonstrating that structure-aware hard negatives can improve temporal-order and mirror-reflection grounding.

Critical strengths
  1. STRIDE evaluates structural grounding across three dimensions: temporal order, mirror reflection, and action identity, with reliability supported by bias controls and human audits.
  2. Motion-side perturbation experiments reveal that standard evaluation metrics can overlook structural mismatches introduced by motion mirroring, highlighting important limitations of existing evaluation protocols.
Critical weaknesses
  1. Structure-aware hard negatives yield inconsistent improvements across models and structural dimensions, with limited gains and occasional performance degradation on Action Identity.
  2. The proposed explanation linking weak structural grounding to conventional contrastive training remains insufficiently validated, as the hard-negative experiments do not fully isolate the underlying causal factors.
Questions for authors
  1. How sensitive are the proposed Δ and GPR metrics to the selection of unrelated motions? Would using different unrelated-motion samples affect the evaluation results?
  2. Why does the Action Identity dimension contain substantially fewer samples than Temporal Order and Mirror Reflection? How does this imbalance affect the statistical reliability of cross-dimension comparisons?
  3. Could the authors provide a more direct comparison with CAR's temporal-order evaluation protocol to demonstrate the additional value of STRIDE's bias controls?
3 comments
AuthorsOpenReview ↗

Response to reviewer bCEU

Thank you for the suggestions. We address control robustness, sample balance, CAR, and training below.

Q1: How sensitive are the proposed Δ and GPR metrics to the selection of unrelated motions? Would using different unrelated-motion samples affect the evaluation results?

A1: Thank you for the question. Different controls change individual scores but preserve the diagnosed structural deficits in all ten rounds of unrelated-motion resampling. Revised Table 3 and Appendix F.2 document the analysis. Because Δ and GPR depend on the unrelated-motion baseline, we test whether its variation changes the diagnosis across all six evaluators and 11 settings. We resampled unrelated motions ten times, excluding each matched source and its mirrors, and report mean ± SD. TO/MR/AI/RC denote Temporal Order/Mirror Reflection/Action Identity/Random Content.

Table 3: Control-motion resampling on HumanML3D and SnapMoGen.

EvaluatorTO ΔMR ΔAI ΔRC ΔMacro GPR (%)
HumanML3D
HumanML3D evaluator0.390 ± 0.0150.013 ± 0.0240.744 ± 0.0150.880 ± 0.00566.8 ± 1.4
TMR0.412 ± 0.0160.430 ± 0.0160.777 ± 0.0200.911 ± 0.00272.0 ± 1.5
CLaM0.495 ± 0.0120.443 ± 0.0190.860 ± 0.0070.955 ± 0.00275.2 ± 1.3
MoBERT0.423 ± 0.0200.179 ± 0.0200.705 ± 0.0190.877 ± 0.00670.2 ± 0.9
ChronRet0.403 ± 0.0230.364 ± 0.0190.852 ± 0.0150.939 ± 0.00372.8 ± 0.6
SnapMoGen
HumanML3D evaluator0.069 ± 0.030-0.004 ± 0.0240.125 ± 0.0400.494 ± 0.01252.1 ± 0.8
TMR0.074 ± 0.0200.204 ± 0.0220.298 ± 0.0370.572 ± 0.01757.8 ± 1.1
CLaM0.120 ± 0.0170.194 ± 0.0160.279 ± 0.0250.687 ± 0.01157.8 ± 0.7
MoBERT0.101 ± 0.0250.089 ± 0.0210.163 ± 0.0260.461 ± 0.01055.4 ± 0.4
ChronRet0.119 ± 0.0170.146 ± 0.0240.207 ± 0.0510.635 ± 0.01357.6 ± 1.4
SnapMoGen evaluator0.928 ± 0.0020.444 ± 0.0140.668 ± 0.0200.995 ± 0.00074.3 ± 0.7

Every assignment preserves structural Δ below RC in all 11 settings. SnapMoGen maintains the TO–MR gap (0.928 vs. 0.444); Macro GPR SDs are 0.4–1.5 percentage points. Individual estimates vary, but the diagnostic pattern does not.


W1: Structure-aware hard negatives yield inconsistent improvements across models and structural dimensions, with limited gains and occasional performance degradation on Action Identity.

W2: The proposed explanation linking weak structural grounding to conventional contrastive training remains insufficiently validated, as the hard-negative experiments do not fully isolate the underlying causal factors.

A2: Thank you for highlighting the need to isolate the training effect. Structural supervision improves mean Δ over matched-budget Vanilla FT in all nine settings, supporting the intervention. Revised Table 5 and Appendix E.3 provide this comparison.

Our hypothesis is that coarse negatives can be distinguished by action content without resolving order or left/right, while structural negatives require these distinctions. The comparison uses the same positive data and training budget, with one final checkpoint across all axes. We report native-dataset mean ± SD over five seeds. Each Δ averages ten rounds of unrelated-motion resampling.

Table 5: Unified five-seed training comparison.

Model / configurationTO ΔMR ΔAI ΔRC ΔRetrieval MedR
HumanML3D evaluator0.3900.0130.7440.88077.0
+ Vanilla FT0.421 ± 0.0190.005 ± 0.0270.721 ± 0.0080.907 ± 0.00766.0 ± 1.5
+ Neg Suite0.559 ± 0.0070.104 ± 0.0110.787 ± 0.0360.914 ± 0.00664.3 ± 1.0
TMR0.4120.4300.7770.91125.0
+ Vanilla FT0.417 ± 0.0060.433 ± 0.0110.772 ± 0.0090.905 ± 0.00523.7 ± 0.3
+ Neg Suite0.494 ± 0.0050.439 ± 0.0080.786 ± 0.0090.921 ± 0.00223.0 ± 0.6
SnapMoGen evaluator0.9280.4440.6680.9951.0
+ Vanilla FT0.923 ± 0.0060.444 ± 0.0080.632 ± 0.0080.992 ± 0.0001.0 ± 0.0
+ Neg Suite0.929 ± 0.0020.451 ± 0.0020.675 ± 0.0160.995 ± 0.0041.0 ± 0.0

Higher mean Δ in all nine settings, with RC/retrieval preserved, supports the added benefit of structural supervision beyond further optimization alone.

AuthorsOpenReview ↗

Response to reviewer bCEU

Q2: Why does the Action Identity dimension contain substantially fewer samples than Temporal Order and Mirror Reflection? How does this imbalance affect the statistical reliability of cross-dimension comparisons?

A3: Thank you for this question. AI’s stricter replacement requirements yield fewer items, but equal-size tests preserve the diagnosis. Revised Appendix F.1, Tables 11–14, reports both checks. Replacements must be visually distinct, mutually exclusive, and preserve other content; duplicate rejection, likelihood filtering, and balancing both directions further limit pairs.

Equal-size test. Equal-size results show that category imbalance does not change our diagnostic conclusion. We matched TO/MR/RC to AI’s sample count (112 HumanML3D; 204 SnapMoGen) and report mean Δ across ten draws in all 11 settings.

Table 11: Equal-size evaluation — HumanML3D (112 per axis).

EvaluatorTO ΔMR ΔAI ΔRC Δ
H3D eval.0.4220.0580.7440.873
TMR0.4040.4270.7770.909
CLaM0.5050.4530.8600.958
MoBERT0.4300.2190.7050.884
ChronRet0.3890.3930.8520.954

Table 12: Equal-size evaluation — SnapMoGen (204 per axis).

EvaluatorTO ΔMR ΔAI ΔRC Δ
H3D eval.0.0630.0020.1250.468
TMR0.0630.2290.2980.586
CLaM0.1240.1990.2790.707
MoBERT0.0830.1190.1630.440
ChronRet0.1350.1430.2070.627
Snap eval.0.9190.4660.6680.989

Every equal-size draw retains structural Δ below RC. SnapMoGen also retains its mean TO–MR gap. Thus, unequal category counts do not affect the diagnostic pattern.

Estimation uncertainty. AI estimates are less precise, but CIs and equal-size tests still support the statistical reliability of our main diagnosis. We report 95% bootstrap CIs, grouping related items by source motion.

Table 13: HumanML3D Δ — 95% confidence intervals.

EvaluatorTO ΔMR ΔAI Δ
H3D eval.[0.322, 0.455][-0.052, 0.074][0.605, 0.862]
TMR[0.336, 0.486][0.367, 0.487][0.625, 0.902]
CLaM[0.432, 0.554][0.381, 0.502][0.755, 0.946]
MoBERT[0.344, 0.491][0.109, 0.243][0.550, 0.834]
ChronRet[0.325, 0.467][0.303, 0.421][0.725, 0.945]

Table 14: SnapMoGen Δ — 95% confidence intervals.

EvaluatorTO ΔMR ΔAI Δ
H3D eval.[-0.009, 0.145][-0.055, 0.043][0.056, 0.196]
TMR[0.016, 0.130][0.140, 0.265][0.155, 0.431]
CLaM[0.046, 0.192][0.127, 0.258][0.127, 0.415]
MoBERT[0.041, 0.160][0.037, 0.143][0.056, 0.265]
ChronRet[0.053, 0.179][0.074, 0.214][0.056, 0.350]
Snap eval.[0.900, 0.952][0.378, 0.504][0.552, 0.774]

This uncertainty coexists with the stable structural-versus-RC ordering in the equal-size test.


Q3: Could the authors provide a more direct comparison with CAR's temporal-order evaluation protocol to demonstrate the additional value of STRIDE's bias controls?

A4: Thank you for suggesting this comparison. STRIDE’s controls distinguish caption preference from motion-grounded improvement that accuracy alone cannot separate. Revised Appendix F.4, Tables 21–22, compares both protocols. We hold captions and scores fixed and add unrelated-motion baselines on STRIDE and CAR’s construction.

On STRIDE: we apply CAR’s caption-choice rule to 953 HumanML3D TO items and add an unrelated-motion baseline. TO Bias measures original-caption preference under this baseline.

Table 21: CAR-style evaluation on STRIDE items.

EvaluatorCAR-style accuracy (%)TO Bias (%)TO Δ
H3D eval.62.438.30.390
TMR70.449.70.412
CLaM72.645.70.495
MoBERT72.652.50.423
ChronRet69.849.30.403

CLaM and MoBERT both score 72.6%, but differ in Bias and Δ. Their identical accuracy therefore does not imply equal reliance on motion.

On CAR: we evaluate original/CAR-trained checkpoints on 2,677 test items under CAR’s official protocol, adding unrelated-motion controls without STRIDE likelihood filtering.

Table 22: Original and CAR-trained checkpoints on CAR items.

CAR checkpointCAR accuracy (%)TO Bias (%)TO Δ
Released original checkpoint68.250.50.356
Released CAR-trained checkpoint99.076.90.956

CAR’s 99.0% accuracy coexists with 76.9% unrelated-motion preference. Its Δ of 0.956 still shows strong motion-grounded improvement. Controls thus identify both a language prior and genuine grounding, rather than treating either as the whole explanation.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer bCEU,

We sincerely thank you for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request you to review our rebuttal. We have also uploaded a revised PDF of the paper and its appendix, addressing your concerns to the best of our ability. We are happy to address any remaining concerns.

Thank you again for your time and effort!

Regards, Authors

Reviewer_HCHkOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

STRIDE is a benchmark examining whether motion-language evaluators distinguish temporal order, left–right differences, and action identity, rather than only overall action content. It comprises 5,869 triples constructed from HumanML3D and SnapMoGen. Language-likelihood filtering reduces item-level bias, while unrelated-motion controls measure evaluator-specific caption preferences. Evaluating the same caption pair with matched and unrelated motions measures the additional discrimination provided by the motion.

The paper analyzes six released evaluators and shows cases in which conventional retrieval and distributional metrics change little after motions are mirrored. It also introduces a training intervention using structural hard negatives. The main contribution is the diagnostic methodology and empirical characterization of structural grounding in existing evaluators.

Critical strengths

1. The evaluation separately controls for dataset-level language bias and evaluator-specific caption preferences.

Likelihood filtering alone does not eliminate evaluator-specific preferences. Section 2.3 measures these preferences by pairing the same captions with unrelated motions and distinguishes the gains provided by matched motions through Δ and GPR. This is an important design choice that reduces the risk of interpreting caption-selection accuracy as structural grounding.

2. Direct motion-side interventions expose errors missed by conventional metrics.

In Table 3, mirroring changes R@1 only from 0.529 to 0.523, while MM-Dist slightly improves from 2.914 to 2.904. This provides concrete evidence that standard evaluation can fail to expose left–right correspondence errors.

In Table 2, the SnapMoGen evaluator achieves a Δ of 0.927 on Temporal Order but 0.435 on Mirror Reflection. The analysis is also useful in showing which structural distinctions individual evaluators can resolve and which they miss.

My weak acceptance is based primarily on the diagnostic methodology and empirical findings; the hard-negative training intervention provides supporting evidence for a possible improvement direction.

Critical weaknesses

Update after author responses (16 Nov 2026): The control-motion resampling, audit documentation, source-motion-level data separation, checkpoint clarification, and matched-budget five-seed results address the concerns below. The intervention’s evaluator-dependent benefits and remaining limitations are now appropriately scoped. I have updated Soundness from 2 to 3.

1. The sensitivity of the main diagnosis to control-motion selection and item quality is unclear.

Appendix D.2 describes unrelated-motion selection for the VLM experiments, but it is unclear whether the same rule applies to the main evaluation of released evaluators. Evidence on the stability of Δ and GPR under different control-motion selections would help assess how strongly the diagnosis depends on that choice.

Appendix B describes human audits and reports a 56.3% text-only preference rate, but does not provide audit sample counts by structural axis or revision and exclusion rates. These details are needed to assess how residual language bias and item ambiguity may affect the main conclusions.

2. The intervention's training-data separation needs clarification.

Section 2.2 states that STRIDE uses the official test splits. Appendix E provides the losses and optimization settings, but does not specify which source splits were used to construct the hard-negative training data or how STRIDE test motions and their associated motion-caption pairs were excluded. Table 6 already identifies NegTMR-A and NegSnap-A as separate Action Identity checkpoints, with the other NegTMR/NegSnap results coming from pooled checkpoints. Reporting Action Identity performance for those same pooled checkpoints, if available, would provide supplementary evidence about the intervention across all three axes. These details matter for interpreting the intervention results; the validity of the preceding diagnostic experiments carries greater weight in my assessment.

The main factors that could change my assessment are item validity and the robustness of the unrelated-motion controls. I would lower my rating if the central diagnosis depends substantially on a particular control selection or ambiguous items.

Questions for authors

Q1. My main priorities are the unrelated-motion controls and item validity. Please describe the control-motion selection rule used for the six released evaluators and provide the human-audit sample counts by axis, the number of annotators per item, and the revision or exclusion rates. Existing protocol descriptions and audit records can address these reporting questions. If available, please also report whether the main structural results remain stable when control motions are resampled.

Q2. Section 2.2 states that STRIDE uses the official test splits. Which source splits were used to construct the hard-negative training data, and how were STRIDE test motion IDs and their associated motion-caption pairs excluded? Table 6 already distinguishes the separate Action Identity checkpoints from the pooled checkpoints. As supplementary evidence, if available, please report the Action Identity results for the same pooled checkpoints used in the other columns.

6 comments
AuthorsOpenReview ↗

Response to reviewer HCHk

Thank you for the constructive feedback. We have clarified the control-selection and audit protocols, verified training-data separation, and added a five-seed training comparison. We address each concern below.

W1: The sensitivity of the main diagnosis to control-motion selection and item quality is unclear. Appendix D.2 describes unrelated-motion selection for the VLM experiments, but it is unclear whether the same rule applies to the main evaluation of released evaluators. Evidence on the stability of Δ and GPR under different control-motion selections would help assess how strongly the diagnosis depends on that choice.

Q1: If available, please also report whether the main structural results remain stable when control motions are resampled.

A1: Thank you for raising this concern. The main diagnosis remains stable across ten rounds of control-motion resampling. We report this in revised Table 3 and Appendix F.2. Individual scores can vary because changing the control changes the estimated caption-preference baseline; we test whether this alters the structural-versus-RC pattern. The released evaluators follow Appendix D.2: controls come from the same dataset and axis but have different motion IDs. Resampling also excludes mirrors of the matched source. The table reports mean ± standard deviation across ten resampling rounds. TO/MR/AI/RC denote Temporal Order/Mirror Reflection/Action Identity/Random Content.

Table 3: Control-motion resampling results on HumanML3D and SnapMoGen.

EvaluatorTO ΔMR ΔAI ΔRC ΔMacro GPR (%)
HumanML3D
HumanML3D evaluator0.390 ± 0.0150.013 ± 0.0240.744 ± 0.0150.880 ± 0.00566.8 ± 1.4
TMR0.412 ± 0.0160.430 ± 0.0160.777 ± 0.0200.911 ± 0.00272.0 ± 1.5
CLaM0.495 ± 0.0120.443 ± 0.0190.860 ± 0.0070.955 ± 0.00275.2 ± 1.3
MoBERT0.423 ± 0.0200.179 ± 0.0200.705 ± 0.0190.877 ± 0.00670.2 ± 0.9
ChronRet0.403 ± 0.0230.364 ± 0.0190.852 ± 0.0150.939 ± 0.00372.8 ± 0.6
SnapMoGen
HumanML3D evaluator0.069 ± 0.030-0.004 ± 0.0240.125 ± 0.0400.494 ± 0.01252.1 ± 0.8
TMR0.074 ± 0.0200.204 ± 0.0220.298 ± 0.0370.572 ± 0.01757.8 ± 1.1
CLaM0.120 ± 0.0170.194 ± 0.0160.279 ± 0.0250.687 ± 0.01157.8 ± 0.7
MoBERT0.101 ± 0.0250.089 ± 0.0210.163 ± 0.0260.461 ± 0.01055.4 ± 0.4
ChronRet0.119 ± 0.0170.146 ± 0.0240.207 ± 0.0510.635 ± 0.01357.6 ± 1.4
SnapMoGen evaluator0.928 ± 0.0020.444 ± 0.0140.668 ± 0.0200.995 ± 0.00074.3 ± 0.7

Structural Δ remains below RC in every assignment and all 11 settings. SnapMoGen retains its TO–MR gap (0.928 vs. 0.444), while Macro GPR standard deviations are 0.4–1.5 percentage points. Thus, the main diagnosis is stable under the tested control selections.


W1: Appendix B describes human audits and reports a 56.3% text-only preference rate, but does not provide audit sample counts by structural axis or revision and exclusion rates. These details are needed to assess how residual language bias and item ambiguity may affect the main conclusions.

Q1: provide the human-audit sample counts by axis, the number of annotators per item, and the revision or exclusion rates.

A2: Thank you for requesting these details. We audited 200 items, 50 per category, with five annotators per item; none required revision or exclusion. Revised Appendix B details both stages. They separate text preference from item ambiguity: language can favor one caption even when the motion–caption match is unambiguous. Sampling was random within TO, MR, AI, and RC. The same five annotators assessed every item in both stages.

Without motion, preference for the original/perturbed caption scored 1/0; an equally likely or uncertain response scored 0.5. The mean across items and annotators was 56.3% original-caption preference.

With motion, annotators checked fluency, whether only the intended structural element changed, and whether the distinction was visible. For RC, they checked that the original caption matched and the unrelated caption did not. All five had to agree on item validity. All 200 passed unchanged: zero revisions and zero exclusions in each category.

The audit supports validity of the sampled items, not the absence of language cues. Residual preference is why we retain evaluator-specific unrelated-motion controls.

AuthorsOpenReview ↗

Response to reviewer HCHk

W2: Section 2.2 states that STRIDE uses the official test splits. Appendix E provides the losses and optimization settings, but does not specify which source splits were used to construct the hard-negative training data or how STRIDE test motions and their associated motion-caption pairs were excluded.

Q2: Section 2.2 states that STRIDE uses the official test splits. Which source splits were used to construct the hard-negative training data, and how were STRIDE test motion IDs and their associated motion-caption pairs excluded?

A3: Thank you for asking us to clarify data separation. Hard-negative training uses only official training splits, with test source motions and all associated derivatives excluded. Revised Appendix E.2 specifies these rules. HumanML3D and TMR use HumanML3D training data, SnapMoGen uses its training data, while STRIDE uses official test splits.

Caption-level exclusion alone could leave another caption or a mirrored copy of the test motion in training. We therefore split by source-motion ID; all captions, edits, segments, and augmentations inherit it. Excluding a test source excludes all these derivatives. Both the original and supplementary training sets have zero source-motion overlap with validation/test data and STRIDE.


W2: Table 6 already identifies NegTMR-A and NegSnap-A as separate Action Identity checkpoints, with the other NegTMR/NegSnap results coming from pooled checkpoints. Reporting Action Identity performance for those same pooled checkpoints, if available, would provide supplementary evidence about the intervention across all three axes. These details matter for interpreting the intervention results; the validity of the preceding diagnostic experiments carries greater weight in my assessment.

Q2: Table 6 already distinguishes the separate Action Identity checkpoints from the pooled checkpoints. As supplementary evidence, if available, please report the Action Identity results for the same pooled checkpoints used in the other columns.

A4: Thank you for identifying this ambiguity. Original Table 6 already reported pooled checkpoints; its caption incorrectly mentioned NegTMR-A/NegSnap-A despite no corresponding rows or ‡ markers, for which we apologize. Revised Table 5 and Appendix E.3 present a unified five-seed comparison. Using one checkpoint across axes, the new table tests whether gains persist beyond a single run and shows higher mean structural Δ than matched-budget Vanilla FT in all nine settings.

For clarity, we report both checkpoint types associated with original Table 6. The pooled and separate AI results under the original Table 6 protocol are:

Action Identity checkpointAI Acc.AI BiasAI Δ
NegTMR-A ‡\ddagger0.8770.5190.745
NegTMR pooled0.8870.5470.750
NegSnap-A ‡\ddagger0.8240.5150.637
NegSnap pooled0.8330.5290.646

The pooled rows use the same checkpoints as the other axes in original Table 6. The revised manuscript replaces that single-seed table with the unified five-seed comparison below.

Five-seed comparison. One checkpoint per seed tests whether a single model improves across axes, without axis-specific checkpoint selection. Vanilla FT controls for gains from further optimization alone. Neg Suite adds TO/MR/AI hard negatives to Vanilla FT, with the same initialization, positive data, and training budget. One final checkpoint serves all axes and retrieval. We report native-dataset mean ± SD over five training seeds, with Δ averaged over ten control-motion resampling rounds per checkpoint.

Table 5: Unified five-seed training comparison.

Model / configurationTO ΔMR ΔAI ΔRC ΔRetrieval MedR
HumanML3D evaluator0.3900.0130.7440.88077.0
+ Vanilla FT0.421 ± 0.0190.005 ± 0.0270.721 ± 0.0080.907 ± 0.00766.0 ± 1.5
+ Neg Suite0.559 ± 0.0070.104 ± 0.0110.787 ± 0.0360.914 ± 0.00664.3 ± 1.0
TMR0.4120.4300.7770.91125.0
+ Vanilla FT0.417 ± 0.0060.433 ± 0.0110.772 ± 0.0090.905 ± 0.00523.7 ± 0.3
+ Neg Suite0.494 ± 0.0050.439 ± 0.0080.786 ± 0.0090.921 ± 0.00223.0 ± 0.6
SnapMoGen evaluator0.9280.4440.6680.9951.0
+ Vanilla FT0.923 ± 0.0060.444 ± 0.0080.632 ± 0.0080.992 ± 0.0001.0 ± 0.0
+ Neg Suite0.929 ± 0.0020.451 ± 0.0020.675 ± 0.0160.995 ± 0.0041.0 ± 0.0

All nine mean structural Δ values exceed Vanilla FT, with RC and retrieval preserved or improved. This supports gains from structural supervision rather than extra optimization alone.

Reviewer_HCHkOpenReview ↗

Follow-up on the control-resampling and training analyses

Thank you for the additional analyses. They appear to address the principal methodological concerns in my initial review.

The ten rounds of control-motion resampling directly examine the sensitivity of the diagnosis to the unrelated-motion assignment. The reported structural-versus-Random-Content pattern persists across all 11 evaluator–dataset settings. In particular, the native SnapMoGen evaluator retains a large Temporal Order–Mirror Reflection gap, with mean Δ values of 0.928 and 0.444. This supports the stability of the main finding under the tested control selections.

The audit description now provides the missing sample counts, annotator coverage, and revision outcomes: 200 items, 50 per category, with five annotators per item. The distinction between residual text preference and motion-grounded item validity is appropriate. The source-motion-level exclusion rule, including associated captions, segments, edits, and augmentations, also answers my question about the separation of intervention-training data from STRIDE evaluation data.

Thank you for correcting the caption of the original Table 6. The clarification that its numerical results already used pooled checkpoints, together with the separately reported Action Identity results, resolves the ambiguity that prompted my question.

The five-seed comparison against matched-budget Vanilla FT strengthens the intervention evidence. Using one final checkpoint per seed across all axes and retrieval avoids axis-specific checkpoint selection.

The additional experiments strengthen the paper’s central diagnosis that aggregate matching metrics can conceal structural grounding failures. The hard-negative intervention also demonstrates that targeted supervision can help, but its ability to mitigate the identified weaknesses remains uneven. All nine reported mean structural Δ scores improve over matched-budget Vanilla FT, yet several gains are modest. Mirror Reflection is particularly relevant here because it is a recurring weakness highlighted by the paper: Δ increases from 0.433 to 0.439 for TMR and from 0.444 to 0.451 for SnapMoGen, with both remaining well below their corresponding Random Content scores. The diagnostic findings are now more firmly supported. The small incremental gains in these two Mirror Reflection settings indicate that the intervention’s demonstrated benefit remains limited in those settings, even though larger improvements are observed elsewhere. This qualifies the scope of the intervention’s effectiveness rather than the validity of the diagnostic benchmark.

Could you clarify which weaknesses the intervention meaningfully alleviates and which remain largely unresolved, and how this distinction is reflected in the revised discussion of its contribution?

Comments on the paper

2 comments

Dear reviewers,

Thank you for your reviews and constructive feedback.

The authors have submitted their rebuttal. Please take a moment to review it and consider whether your concerns have been adequately addressed. It would be helpful if you could briefly indicate:

  • Whether the rebuttal addresses your main concerns;

  • Whether any significant issues remain unresolved;

  • Whether your overall recommendation has changed.

If appropriate, please also update your review and recommendation in the system.

Thank you for your time and valuable contributions.

Best regards,

AC

AuthorsOpenReview ↗

General Rebuttal Comment by Authors

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their careful evaluation and constructive feedback. We particularly appreciate the following positive observations from the reviewers:

  • "addresses an important limitation of motion-language evaluation: conventional metrics may overlook fine-grained spatial and temporal relationships." (Reviewer aCwJ)
  • "The evaluation separately controls for dataset-level language bias and evaluator-specific caption preferences." (Reviewer HCHk)
  • "reveal that standard evaluation metrics can overlook structural mismatches" and "highlighting important limitations of existing evaluation protocols." (Reviewer bCEU)

During the discussion period, we strengthened the manuscript through additional experiments and methodological clarifications. We summarize the main updates below.

1. Robustness of the diagnostic findings — addressing HCHk, bCEU, and aCwj

  • We conducted ten rounds of control-motion resampling. The diagnostic pattern persists throughout. Additional analyses of semantic exclusion, multiple controls per item, and bidirectional consistency further support the central diagnosis under alternative evaluation conditions (Table 3 and Appendices F.2–F.3).

2. Benchmark reliability and statistical validation — addressing HCHk, bCEU, and aCwj

  • We expanded the human-audit documentation with sample counts, annotator coverage, and revision/exclusion outcomes, distinguishing text-only preference from motion-visible item validity. We explained the smaller Action Identity category and added equal-size evaluations, bootstrap confidence intervals, and stricter language filtering. These analyses support benchmark reliability and show that balancing category sizes and tightening text-bias controls preserve the main diagnostic pattern (Appendix B and Appendix F.1).

3. Controlled intervention evidence and generalization — addressing HCHk, bCEU, and aCwj

  • We added a matched-budget, five-seed comparison against ordinary contrastive fine-tuning(Vanilla FT). This comparison isolates the added benefit of structural supervision: mean structural Δ improves over vanilla FT.(Table 5).

  • The revised discussion clarifies the uneven benefits: larger gains in Temporal Order and Mirror Reflection for the HumanML3D evaluator and in Temporal Order for TMR, improved mean Action Identity Δ across all three evaluators, and limited Mirror Reflection gains for TMR and SnapMoGen. We further evaluated held-out within-axis variations and analyzed cross-dataset composition effects (Table 5, Section 4, and Appendices E.3–E.4).

  • We also clarified source-motion-level training/test separation and resolved the original table-caption ambiguity (Appendix E.2 and our response to HCHk).

4. Clearer positioning and direct comparison with CAR — addressing bCEU and aCwj

  • We added direct comparisons with CAR to demonstrate the information supplied by unrelated-motion controls beyond caption-choice accuracy. We expanded the discussion of ViLMA and VeMo, clarifying STRIDE’s distinctive focus on measuring the contribution of motion after accounting for residual caption preferences (Section 5 and Appendix F.4).

5. Practical evaluation using generated motions and human judgments — addressing aCwj

  • We generated 90 motions covering all three structural axes and obtained judgments from five annotators. Evaluation on the 80 motions with exclusive caption labels demonstrates the practical value of identifying structural evaluation errors and shows improved evaluator–human agreement in specific settings (Table 6 and Appendix G).

We appreciate the encouraging follow-up feedback during the discussion, including the observations that the additional analyses “appear to address the principal methodological concerns,” that “the new experiments have improved the paper,” and that the results offer “strong evidence in support of their claims,” alongside an expressed willingness to raise the score.

The revisions, highlighted in blue in the revised PDF, reinforce our central insight: strong overall motion–language matching does not necessarily imply fine-grained structural grounding. By separating caption preferences from motion-grounded discrimination, STRIDE provides the community with a diagnostic framework for identifying evaluator weaknesses that conventional metrics can miss. We hope it supports more reliable evaluation and guides the development of models that better capture temporal order, left–right relations, and action identity.

We believe the additional evidence and clarifications address the concerns raised during review. Thank you for your time and consideration.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.