acceptodds
Under review as a conference paper at ICLR 2027

Weak Models or Insensitive Metrics? Auditing and Training Fine-Grained Motion Captioning

Abstract

Motion captions are evaluated by n-gram overlap, embedding similarity, or an LLM judge that scores the caption as a whole, but whether these metrics notice one wrong motion fact, such as the acting side or the direction of movement, has not been tested attribute by attribute. Without that test, a weak model cannot be distinguished from an insensitive metric. We build MotionADV, a Chinese–English challenge set of caption pairs that differ in exactly one fact, from 494 clips whose captions were corrected by professional motion artists. Under a single wrong fact, n-gram and embedding-based metrics move less than a fifth of the way to an unrelated caption, and n-gram metrics even prefer the wrong caption to a fact-preserving paraphrase. A whole-caption LLM judge cannot both detect and localize the error. We define MotionFact, a fixed rubric of motion attributes over a typed action schema, and build on it MotionJudge, a decomposed evaluation protocol that checks each attribute against a reference with its own questions. With Qwen3.5-27B, MotionJudge reaches a detection rate of at least 0.90 and a pass rate of at least 0.89 on every attribute, and an ablation that adds the rubric and then the schema-based protocol shows what each contributes. We further train Motion-VLM, a general video-language model that reads multi-view renders and motion tokens, and states MotionFact facts before its caption. Under both judges, Motion-VLM outperforms five published motion captioners on HumanML3D and demonstrates superior cross-dataset transferability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Motion Understanding, LLM, VLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EwLWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether weak fine-grained motion-caption scores come from the models or from metrics that cannot see a single wrong fact. It builds MotionADV, minimal caption pairs differing in exactly one fact from 494 artist-corrected clips, and shows that n-gram and embedding metrics barely react to the error, and that n-gram metrics even prefer the wrong caption to a fact-preserving paraphrase. It then contributes MotionFact, an eleven-attribute rubric over a typed action schema, MotionJudge, a decomposed protocol that checks each attribute against a reference and, with Qwen3.5-27B, reaches at least 0.90 detection and 0.89 pass on every attribute, and Motion-VLM, a Qwen3.5-4B fine-tune on multi-view renders, motion tokens, and fact-first targets that outperforms five published captioners on HumanML3D.

Critical strengths

1. The audit quantifies a blind spot in motion-caption metrics On minimal pairs built from 494 artist-corrected clips, a single wrong fact moves every similarity metric less than a fifth of the way toward an unrelated caption, and n-gram metrics rank the wrong caption above a fact-preserving paraphrase in at least 1,258 of 1,261 Chinese pairs. This matters because standard scores cannot certify the physical facts of a caption, which is the gap fact-level evaluation must close.

2. The results come with the controls and records needed to check them The paper reports ceiling, floor, and repeat-noise controls, a label-leakage subset excluding the judge's base model, a replication under a second base model, an explicit accounting of its own misses on real errors, and the release of MotionADV with per-sample validator votes. This matters because the challenge set and its analyses can be checked and reused step by step, making them a durable resource for fact-level evaluation.

Critical weaknesses

1. MotionJudge lacks a human anchor and its validity chain runs through one model family The attribute edits of MotionADV are generated by Qwen3.5-27B, the labels are validated by three language models with escalated cases decided by GPT-5.5, and MotionJudge is built on Qwen3.5-27B, so one family generates the edits, contributes two of the three label validators, and supplies the judge base. The paper itself raises the risk that the judge shares blind spots with the labels, and the labels are filtered by models rather than humans. Two mitigations are reported. Detection stays at or above 0.94 on the subset whose labels pass without the Qwen vote, and gemma as a second base model reproduces the ranking, but several attributes fall below the paper's own thresholds of 0.90 detection and 0.89 pass, with bilateral-mode detection at 0.888 and 0.831 and motion-type pass at 0.833 and 0.788. On real errors the paper reports contradicted rates of only 30% and 20% for side and direction rewrites and lists 22 misses, 13 caused by alignment rejecting a matching action and the rest by a post-rule reassignment, a value read that misses the side, or a reference parse with no usable fact. The grader assigns each miss a score of 2, but the attribute-level verdicts remain unreliable on real errors, and that verdict level is the stated advantage over a whole-caption judge. MotionJudge has not been validated against human raters, so the replacement for the metrics the paper calls unreliable has no validity evidence independent of models, and both the audit and the model conclusions rest on an instrument validated by models alone.

2. The model-side claim rests on self-built judges, and the fact-level evidence shows coverage rather than accuracy The advantage of Motion-VLM over the strongest published baseline, MG-MotionLLM, is only +0.057 on MotionJudge (2.483 against 2.421) and +0.033 on the whole-caption judge, both of which are built in this paper and share the Qwen family with Motion-VLM, and the gemma replication keeps the ranking but does not meet the same reliability thresholds. The authors' own summary-only baseline already matches MG-MotionLLM, so the structured target and the motion tokens contribute only +0.061 and +0.012 over plain fine-tuning. The error rate among stated facts is 12.4% against 12.5% for that baseline, and the +1.5 point gain in supported facts comes mainly with fewer omissions (67.5% to 66.2%) while the contradiction rate barely moves (+0.2), so the model states more facts rather than more accurate ones. On the established R-Precision@1 protocol outside the two self-built judges, MG-MotionLLM still ranks first, and the paper explains this through an encoder-insensitivity finding from its own audit. Because the fact-level precision gain attributed to the model input and training target does not appear in the paper's own comparison, the accuracy claim rests only on in-house instruments, and the second contribution remains unsubstantiated at the mechanism level.

3. The audit is disconnected from the main experiments and its premise is unproven on the main benchmark The paper attributes low scores on motion facts to both weak models and insensitive metrics, but its main model comparisons fall where the two do not disagree. On HumanML3D, BLEU-4 ranks the nine systems exactly as MotionJudge does (Spearman 1.00), ROUGE-L agrees at 0.98, and the single-call whole-caption judge also agrees at 0.98, so the system ranking in the main table can be reproduced with existing metrics and the need for a protocol costing about 17 calls per caption is never demonstrated. The paper explains the agreement by noting that HumanML3D references are short and state few facts, which concedes that the main benchmark cannot exercise the problem the paper sets out to fix. The divergence that changes a conclusion appears in the zero-shot comparison on Motion-X++, and the paper does not quantify how often or how severely metric blindness affects real system comparisons, so the case for replacing the metrics still rests mainly on synthetic probes and the empirical basis for the metric side of the thesis remains weak.

Questions for authors

Has MotionJudge been compared against even a few hundred human adjudications, and how does human agreement vary across attributes? On real errors, how much of the advantage of attribute-level verdicts over the whole-caption judge remains, and do the 30% and 20% contradicted rates on side and direction rewrites change the conclusion that decomposition outperforms whole-caption scoring? Do the attributes that fall below the thresholds under the gemma base change the system ranking of the main table, or only the absolute scores? On HumanML3D or any fact-dense subset, is there any instance where MotionJudge and BLEU-4 order the systems differently?

Please add a small human validation study and report agreement rates, and recalibrate the detection and pass thresholds against it. Re-report the per-attribute results on real errors and under the second base model, and provide a system-level comparison on a fact-dense subset to connect the audit findings to the model results. Split the model claim into separately testable coverage and precision statements, or add conditional precision evidence. Clarify the licensing and release plan for the 494 industrial clips, and state the minimal essential difference from KPM-Bench and MotionHalluc in the main text.

7 comments
AuthorsOpenReview ↗

Response to Reviewer EwLW (Part 1/6 - W1 & Q1)

We thank the reviewer for the careful and detailed review, and for recognizing that the audit quantifies a blind spot that standard scores cannot certify and that the paper reports the controls needed to check it. We address the three weaknesses, the four questions and the requests below, with new experiments for each.


W1: MotionJudge lacks a human anchor, and its validity chain runs through one model family.

Thanks for this careful analysis and the concrete suggestions. In short, MotionJudge agrees with human judgments on real errors and on generated captions, and it does not depend on one model family. Following the suggestions, we report its agreement with the artists' corrections of 494 production drafts per attribute, and we further add a human check on generated captions. We split the concern into four questions:

  • (i) & Q1 Has MotionJudge been compared against human adjudications? Yes. It agrees with professional artists on a few hundred real errors (Table A). On 30 motions where MotionJudge and the whole-caption judge prefer different captions, the artists agree with MotionJudge 20 times and with the whole-caption judge 4 times (Table B).
  • (ii) Does the validity chain depend on one model family? Not on Qwen alone. Every label must also be confirmed by a model outside the Qwen family, and the same protocol also helps GLM and Llama as the judge model (Table C).
  • (iii) & Q3: Does gemma change the ranking? The ranking stays the same. Gemma only lowers the absolute scores (Table D).
  • (iv) & Q2: Do the low contradicted rates on real errors change the conclusion? The conclusion holds. Many rewrites that MotionJudge does not mark as contradicted are ones where the artist replaced the whole action, so “not mentioned” is the right verdict. MotionJudge still catches more of the artists' rewrites than the whole-caption judge (Table E).

The details follow.


W1 - (i) & Q1: Has MotionJudge been compared against human adjudications, and how does agreement vary across attributes?

  • The draft–correction pairs give a few hundred human adjudications per attribute (Table A). Their labels are the edits that professional motion artists made to 494 production drafts after viewing each clip, and code only turns each edit into labels. For each attribute, Table A gives how often MotionJudge agrees with the artist: it should flag a value the artist rewrote or added, and should not mark a value the artist kept as wrong. MotionJudge flags nearly every rewritten action, side and direction and almost every side or direction the artist added, and does not mark most unchanged facts as wrong. Added actions are the weakest case: MotionJudge flags four in five of them, mostly as a wrong action rather than a missing one. The paper's Table 24 counts only flags that call such a fact missing, which is why it reports a much lower rate for actions. Speed, amplitude and count have too few rewrites to report, and ✗ marks rates below the paper's thresholds.
  • A pairwise check where two judges disagree (Table B). On 30 HumanML3D motions where MotionJudge and the whole-caption judge prefer different captions of Motion-VLM and MG-MotionLLM, three professional motion artists watched each motion and agreed on the better caption or “about the same”. Because the two judges always give different answers, the artists can match one judge or neither. They side with MotionJudge on 20 motions and with the whole-caption judge on 4.

Table A. MotionJudge vs. the artists' corrections (494 drafts, Chinese).

AttributeDraft value wrong (artist rewrote it)Draft left the fact out (artist added it)Draft right (artist left it unchanged)
action0.970 (169)0.797✗ (79)0.855✗ (145)
side1.00 (54)0.993 (134)0.983 (239)
direction0.908 (76)0.908 (130)0.897 (117)

Table B. Artists' choice where the two judges disagree (30 HumanML3D motions).

Whose preference matches the artists' choiceMotions
MotionJudge20 (66.7%)
Whole-caption judge4 (13.3%)
Neither: the artists preferred Motion-VLM, which neither judge preferred5 (16.7%)
Neither: the artists preferred MG-MotionLLM, which neither judge preferred1 (3.3%)
AuthorsOpenReview ↗

Response to Reviewer EwLW (Part 2/6 - W1 & Q2, Q3)

W1 - (iii) & Q3: Do the attributes below threshold under gemma change the ranking, or only the absolute scores? (Q3)

They change only the absolute scores. Under gemma-4-31B, Motion-VLM remains first, and its margin over MG-MotionLLM grows (Table D). Gemma flags motion-type errors more often than Qwen, including on some correct captions. This over-flagging lowers every model's score, but not their order. Table D lists the MotionJudge scores, from 0 to 5, under each judge model.

Table D. Ranking under two judge models (HumanML3D).

Qwen3.5-27Bgemma-4-31B
MotionGPT2.3032.047
Motion-Agent2.3752.132
MG-MotionLLM2.4212.154
summary-only2.4202.210
Motion-VLM2.4832.253
Motion-VLM − MG-MotionLLM+0.057 [+0.019, +0.096]+0.100 [+0.058, +0.141]
motion type, detection / pass, Chinese0.950 / 0.9340.983 / 0.833
motion type, detection / pass, English0.928 / 0.9340.952 / 0.788

W1 - (iv) & Q2: On real errors, do the low contradicted rates for side and direction change the conclusion that decomposition helps? (Q2)

Table E compares MotionJudge with the whole-caption judge on the sides and directions that the artists rewrote or left unchanged. A value counts as flagged when the judge marks it as wrong or not fully supported.

  • Many rewrites change the whole action, where “not mentioned” is correct. About a third of the artists' side rewrites and more than half of their direction rewrites replace, add or delete the whole action. Of the rewrites that MotionJudge either marks as contradicted or misses, it marks most as contradicted, and even a miss lowers the caption's overall score to 2 of 5.
  • The attribute-level advantage remains on real errors. We scored the same 494 pairs with the whole-caption judge that names the wrong attributes. MotionJudge flags more of the artists' side and direction rewrites and marks fewer unchanged sides as wrong. The whole-caption judge marks fewer unchanged directions as wrong, but misses a quarter of the direction rewrites.
  • Most misses come from one alignment step, and a simple retry reduces them. They happen because the step that matches caption actions to reference actions refuses to match two actions that differ only in side or direction, so a wrong side is recorded as “not mentioned” instead of wrong. Retrying the match once with the side and direction words hidden (align*) raises the share of rewrites marked as a wrong value from 30% to 41% for side and from 20% to 29% for direction, without changing the other rates. We will adopt it in the revision.

Table E. MotionJudge vs. the whole-caption judge on real errors (494 drafts).

JudgeSide rewritten: flaggedSide unchanged: not marked wrongDirection rewritten: flaggedDirection unchanged: not marked wrong
Whole-caption judge (+Rubric)0.8520.9410.7650.979
MotionJudge1.000.9830.9080.897

We will add Tables A–E and the alignment retry to the revision.

AuthorsOpenReview ↗

Response to Reviewer EwLW (Part 3/6 - W2)

W2: The model-side claim rests on self-built judges, and the fact-level evidence shows coverage rather than accuracy.

Thanks for this suggestion. In short, Motion-VLM's advantage holds without our own judges, and it states more facts at no loss of precision. Following the suggestion, we add a judge-free mirror test and a judge from another family, and we split the claim into coverage and precision. We answer four questions:

  • (i) Does the margin hold without our judges? Yes. Motion-VLM is first on a mirror test that uses no LLM judge (Table F), and also with gemma as the judge.
  • (ii) Does summary-only already match MG-MotionLLM? Yes, because it already uses our three-view input. Our full model is still significantly better than it.
  • (iii) Is the gain coverage or precision? Both. Motion-VLM states more correct facts than every baseline, and on the facts that both models state, it makes no more errors. The extra facts it states are as accurate as the rest (Table H).
  • (iv) Why does MG-MotionLLM lead on R-Precision@1? Because R-Precision does not check facts. It prefers captions with a wrong side, speed or position, it cannot tell left from right, and it even ranks the human references below both models (Tables I and J).

The details follow.

--

W2 - (i) Does the margin over MG-MotionLLM hold without our judges?

  • A mirror test needs no LLM judge (Table F). HumanML3D pairs every test motion with its exact mirror, whose reference swaps left and right. A caption counts as right only if it names the correct side on both. Motion-VLM leads every model, MG-MotionLLM included. Table F gives the share of pairs right on both sides and the difference from Motion-VLM with its 95% interval, which excludes zero for every model.
  • Other metrics and a judge from another family agree. Motion-VLM also leads the n-gram and embedding metrics on HumanML3D (paper Table 2), and it ranks first with gemma as the judge model (Table D).

Table F. Mirror test, no LLM judge (HumanML3D, 660 pairs).

ModelSide right on motion and mirrorMotion-VLM − model
DEMO0.118+0.291 [+0.253, +0.329]
MotionGPT30.189+0.220 [+0.182, +0.259]
MotionGPT0.261+0.148 [+0.109, +0.188]
Motion-Agent0.277+0.132 [+0.097, +0.165]
MG-MotionLLM0.326+0.083 [+0.049, +0.120]
Motion-VLM0.409–

W2 - (ii) Does the summary-only baseline already match MG-MotionLLM?

It does, but it is not plain fine-tuning. Summary-only already uses our three-view, color-coded input, one of our contributions and the largest single gain in our ablations. The full model is still significantly better than summary-only, with either judge model and on the judge-free mirror test (Table G). Both added parts help. The structured target gives a significant gain on HumanML3D. The motion tokens give a large gain on Motion-X++, whose motion data includes hand and finger joints, about twice as large on hand clips, whose reference mentions hands or fingers, as on the others. In Table G, significant means that the 95% interval of the paired difference excludes zero. They add little on HumanML3D, where the render already shows every joint.

Table G. What the full model adds over summary-only.

ChangeBenchmarkMeasureWithout → withSignificant
summary-only → full modelHumanML3DMotionJudge (Qwen)2.420 → 2.483yes
summary-only → full modelHumanML3DMotionJudge (gemma)2.210 → 2.253yes
summary-only → full modelHumanML3Dmirror test0.351 → 0.409yes
adding the structured targetHumanML3DMotionJudge2.420 → 2.465yes
adding the motion tokensMotion-X++MotionJudge1.584 → 1.805yes
↳ on hand clipsMotion-X++MotionJudge1.728 → 2.028yes
↳ on other clipsMotion-X++MotionJudge1.388 → 1.528yes
Reviewer_Naz9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new suite for 3D human motion captioning. It consists of MotionFact, a set of attributes describing human motion, MotionJudge, an evaluation protocol that operates with MotionFact, and a new MotionJudge model that takes in multi-view input to output the MotionFact-style captioning, finetuned from Qwen-4B.

Critical strengths
  • The paper discusses an important topic of human motion captioning, that serves at the core of text-to-motion generation.

  • The paper is clearly presented, and coherent based on the key observations that motion facts come from both weak models and insensitive metrics.

  • Fig 1 is a very clear presentation of different approaches.

  • Sec 3.3 provides an comprehensive study.

  • Cross-dataset transfer is evaluated in Table 4.

Critical weaknesses
  1. Overall, I think the technical contribution is not significant. MotionFact is an intuitive design which is has over-simplified the motions that can be executed by a human, MotionJudge is a LLM-as-a-judge framwork that can be achieved with reasonable amout of prompt engineering, MotionVLM is finetuning a small Qwen model with LoRA. I personally did not gain much more insights or deeper understanding in proposed approach.

  2. It is claimed that an LLM judge can output a verdict but fails to locate where the error is. I think this is an underestimation of the leading LLMs. I think directly feeding MotionFact and MotionJudge into any leading LLM can produce sufficiently good, typesafe results. Cost might be a point of argument that the authors might consider to discuss.

  3. Regarding the input videos which renders the skeletons. Please how does this extends to more refined motion captioning, as it is very difficult to observe smaller keypoints (wrist, hands). Would you consider rending them in SMPL format or 3D characters that might be more aligned with the training distribution of a VLM?

  4. It will be diffcult to argue that Motion-VLM is still a general video-language model after finetuning for motion captioning. To validate this, results on general video understanding benchmarks will be necessary.

Non-critical:

  • Line 132: it would be clearer to what are the three causes, as this only appeared once in the intro.

  • Line 207: it would be clearer to provide some explanation on direction / oscillation-axis errors in motion captioning context.

Questions for authors

Please see weaknesses 1-4. I will raise my score if my concerns are addressed.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Naz9 (Part 1/5)

We thank the reviewer for the careful review, for finding the topic important and the paper clear and coherent, and for the concrete suggestions, which helped us see where the paper was unclear. We address the four weaknesses and the two minor points below, with new experiments for W2–W4.


W1: The technical contribution is not significant, and the approach offers limited new insight.

Thanks for this comment, and we apologize that the paper did not state our contributions clearly enough. Each component is simple in form (a schema of motion facts, an LLM judge, and LoRA on a 4B Qwen model). To clarify them, we split the reviewer's concern into the three questions below, one per component, and add one further experiment, (iv), that connects them. We hope this addresses the reviewer's concern. In short, MotionFact and MotionJudge make fine-grained motion facts measurable, and Motion-VLM shows that a general VLM, measured this way, is a strong motion captioner.

(i) Does MotionFact over-simplify the motions a human can execute?

  • It starts from the audit. Existing captioners and metrics fail on facts as basic as left vs. right and forward vs. backward. For example, MG-MotionLLM states the side correctly on both a motion and its mirror for only about a third of motions (Table A). We therefore first fixed the minimal facts that must be checked to judge a motion caption.
  • It is built on meta-actions. Following movement analysis for dance and animation [1], MotionFact writes a motion as an ordered sequence of meta-actions. Each carries its semantic action, finer refinements of it (motion type, oscillation axis, articulation, bilateral mode) and kinematic facts (side, direction, rotation, speed, amplitude, count), each with a closed set of values that code can compare.
  • It is orthogonal and extensible, following domain-specific language (DSL) design. We agree that it does not describe every motion a human can perform. Each slot changes independently of the others, so one fact can be edited or checked at a time, and a new attribute is added as a new slot without changing the protocol code. The oscillation-axis, articulation, bilateral-mode and compound-direction slots were added this way during development.

[1] Rudolf Laban and Lisa Ullmann. The mastery of movement. ERIC, 1971.

(ii) Can MotionJudge be achieved with a reasonable amount of prompt engineering?

  • Better wording is not enough. We agree that a leading LLM with a careful prompt goes a long way, so we gave one call the same rubric, the same schema and the same comparison code, the strongest single prompt we could write. It still leaves attributes below threshold with every model, and Claude Opus 5.5 flags about one correct caption in five (Tables E and G under W2).
  • The protocol backs up the model. MotionJudge is an evaluation protocol (harness) around the LLM: it changes how the judgment is split, not how it is worded. It asks narrow questions the model answers reliably (align the actions, then read one attribute from one phrase), and code checks every answer. Values outside the closed set are rejected, quotes must appear verbatim, and values are compared by code, so errors are caught at each step instead of being passed on.
  • It transfers across models without re-tuning. With unchanged prompts, the same protocol improves on the single call for every open judge model we tried, from three families (Table G under W2). W2 below measures what the rubric, the schema and the decomposition each add.
AuthorsOpenReview ↗

Response to Reviewer Naz9 (Part 2/5)

Response to Reviewer Naz9 (Part 2/5)

(iii) Is Motion-VLM only LoRA fine-tuning of a small Qwen model?

  • It tests a common assumption. Prior work concluded that general VLMs caption motion poorly, and built dedicated motion tokenizers such as the VQ-VAE codes of MotionGPT and MG-MotionLLM.
  • It needs no motion encoder. Motion-VLM projects raw joint positions linearly into the language model, with no codebook or MotionCLIP-style pretraining, and trains LoRA on a 4B VLM.
  • Multi-view input and the training design address occlusion. A single view can hide a limb or leave left and right ambiguous. Motion-VLM reads three views, and its training target makes it state where each camera stands and list the MotionFact facts of each action, computed from the motion data, before the caption. On HumanML3D, three views instead of one give the largest gain of any design choice (Table B), further improving the VLM's understanding of motion.
  • It outperforms the dedicated motion LLMs, under MotionJudge and on the judge-free mirror test below (Table A). A general VLM, without a dedicated motion tokenizer, can be a strong motion captioner.

Table A. Motion-VLM vs. dedicated motion LLMs on HumanML3D. Motion encoder: how motion enters the language model. Mirror test: share of motions whose side is stated correctly on both the motion and its mirror, scored without any LLM judge.

ModelSizeMotion encoderVideoMotionJudge (0–5)Mirror test
DEMO8BMLP adapter on raw joints–1.9250.118
MotionGPT3238MVAE (continuous latents)–2.2310.189
MotionGPT220MVQ-VAE tokenizer–2.3030.261
Motion-Agent2BVQ-VAE tokenizer–2.3750.277
MG-MotionLLM220MVQ-VAE tokenizer–2.4210.326
Motion-VLM4Bnone (one linear layer)3 views2.4830.409

Table B. What each module adds (MotionJudge, 0–5). The video-only model differs from the one-view model only in the views, from the summary-only model only in the target, and from Motion-VLM only in the motion tokens. The one-view model uses a fixed camera.

ModelViewsStructured targetMotion tokensHumanML3DMotion-X++
one-view model1✓–2.252–
summary-only model3––2.4201.592
video-only model3✓–2.4651.584
Motion-VLM3✓✓2.4831.805

(iv) A further experiment to clarify the contributions: the mirror test.

To help clarify our contributions, we added one experiment that needs no LLM judge. Purpose: to check, on the most basic fact, left vs. right, whether captioners and metrics get it right. Result: existing captioners and the standard retrieval metric largely fail, MotionJudge does not, and each design choice of Motion-VLM helps. Design: HumanML3D pairs every test motion with its exact mirror, whose reference swaps left and right. A caption counts as right only if it names the correct side on both, so guessing or always naming one side scores zero.

  • Captioners: it confirms the audit. The strongest motion LLM, MG-MotionLLM, gets both sides right on only about a third of pairs, and Motion-VLM is the best of all models (Table A).
  • Metrics: it shows the blind spot that MotionFact and MotionJudge address. The R-Precision evaluator cannot tell a motion from its mirror: given a reference that names a side, it places the motion closer than its mirror only at chance. MotionJudge checks the side as a typed fact and accepts a wrong side in only 5 of 107 cases (Table C).
  • Model: each design choice of Motion-VLM helps. Three views, the structured target and the motion tokens each raise the mirror score (Table D).

Together, these answer our title question: errors in fine-grained motion captions come from both weak models and insensitive metrics.

Table C. Can the metric tell left from right? (HumanML3D)

MetricTestResult
R-Precisionmotion vs. its mirror: which fits a reference that names a side?50.2% right (chance)
R-Precisionleft and right swapped in every reference0.455 → 0.455 (no change)
MotionJudgecaptions with the wrong side: accepted as correct4.7% (5 of 107)

Table D. Mirror test for each module (HumanML3D, share of mirror pairs with the side right on both, no LLM judge). Same models as Table B.

ModelViewsStructured targetMotion tokensMirror test
one-view model1✓–0.268
summary-only model3––0.351
video-only model3✓–0.377
Motion-VLM3✓✓0.409

We will add these contribution statements and the mirror test to the revision.

AuthorsOpenReview ↗

Response to Reviewer Naz9 (Part 3/5)

Response to Reviewer Naz9 (Part 3/5)

W2: A leading LLM fed with MotionFact and MotionJudge may produce sufficiently good, type-safe results, and cost may be worth discussing.

Thanks for this insightful suggestion. We tested it directly, including the cost. We agree that a frontier LLM finds wrong values well on its own, and with MotionFact's schema it also catches omitted facts. In one call, however, it still flags correct captions, while MotionJudge passes every attribute with an open model, at lower cost and reproducibly. We ran the full MotionADV in both languages with the frontier API model Claude Opus 5.5, and with three open models from three other families. The results are shown below.

Table E. A frontier LLM vs. MotionJudge on MotionADV (Chinese / English). ✗: below threshold.

JudgeJudge modelAttributes below thresholdCorrect captions not flagged*Cost / 1,000 captions
One verdictClaude Opus 5.510 / 100.993$1.9
+RubricClaude Opus 5.59 / 90.988$2.6
+Schema (1 call)Claude Opus 5.51 / 20.794✗$11.0
MotionJudgeQwen3.5-27B (open)0 / 00.976≈16 H100-min

*Chinese captions with no error: rewordings and word-for-word copies of the reference. A judge should flag none.

(i) A whole-caption judge catches wrong values but misses omitted facts, Claude included. If the reference says “raises the left hand” and the caption says “raises a hand”, an animator sees at once that the side is missing, but the judge does not treat it as important and accepts the caption. Claude finds almost every swapped side, left written as right, but almost none of the removed ones (Table F). A metric like this rewards captions that leave the side out. This is the main reason it leaves most attributes below threshold with the rubric alone.

Table F. Side errors detected, by kind (Chinese / English). ✗: below the 0.90 detection threshold.

JudgeLeft ↔ right swappedSide word removed
Claude, One verdict96.8% / 95.7%0.0%✗ / 12.9%✗
Claude, +Rubric90.5% / 90.5%17.4%✗ / 15.8%✗
Claude, +Schema (1 call)92.1% / 92.2%98.3% / 97.0%
MotionJudge (Qwen3.5-27B)99.2% / 98.2% of all side errors

MotionJudge misses 2 of 241 Chinese and 4 of 217 English side errors in total, so it detects at least 98% / 96% of each kind.

(ii) MotionFact tells the model exactly which facts to check. Every action and every limb has its own slots, so an omitted side shows up as an empty slot rather than going unnoticed. With the schema, Claude catches almost every removed side and leaves only one or two attributes below threshold (Tables E–F). This per-action, per-limb check is what MotionFact was designed to provide.

(iii) In a single call, the frontier model also flags correct captions. On captions that are word-for-word identical to their reference, where no error should be reported, +Schema (1 call) with Claude reports an error on about one in five, almost always by reading a different value from the same words (Table E). MotionJudge catches both kinds of side error while rarely flagging correct captions (Tables E–F). With the open models, the single call also leaves more attributes below threshold than the same model under MotionJudge (Table G).

Table G. The same protocol on open models from three families: attributes below threshold (Chinese / English).

Judge model+Rubric+Schema (1 call)MotionJudge
Qwen3.5-27B8 / 73 / 50 / 0
GLM-4-32B9 / 99 / 93 / 3
Llama-3.3-70B7 / 68 / 84 / 6

(iv) As the reviewer suggests, cost matters: MotionJudge is cheaper and reproducible. At list price, +Schema (1 call) with Claude costs several times as much per caption as the whole-caption judges, and its answers depend on a hosted model that changes over time. MotionJudge runs on one open model on a single H100 GPU, offline and deterministically (temperature 0, frozen weights), which we think is important for a metric that others need to reproduce.

We will add Tables E–G and this cost discussion to the revision.

Reviewer_XnxUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether motion-caption metrics detect fine-grained factual errors. It introduces MotionADV, a challenge set of controlled caption edits, and MotionJudge, an attribute-level evaluation protocol based on a typed motion schema. It also trains Motion-VLM with multi-view renders, motion tokens, and structured targets, and evaluates its captioning performance on HumanML3D and Motion-X++.

Critical strengths
  • The controlled edits and fact-preserving rewrites provide a clear test of whether metrics distinguish factual errors from harmless wording changes. Reporting detection and pass rates separately makes the audit informative.
  • MotionJudge provides explicit attribute-level feedback and distinguishes contradictions from omissions. The ablations show where the decomposed protocol improves over a whole-caption rubric.
  • The model experiments are thorough. Removing, zeroing, and swapping motion tokens help establish their contribution, and the paper clearly reports that their benefits differ across datasets.
Critical weaknesses
  1. The evaluation does not isolate the benefit of multiple LLM calls. A single-call baseline using the same rubric, action schema, and deterministic comparison is missing. The thinking settings are also unspecified, making the accuracy–cost trade-off difficult to assess.
  2. MotionJudge has not been validated against human judgments of generated captions. The controlled-edit audit is useful, but it does not establish whether the metric’s preferences on natural model outputs agree with people who inspect the motion.
Questions for authors
  1. Could you compare MotionJudge with a single-call baseline using the same rubric, action schema, and deterministic comparison? Please clarify the thinking settings and report accuracy, latency, and token usage under comparable inference settings.
  2. Could you provide a small blind human comparison of generated captions, particularly where MotionJudge and the whole-caption judge disagree, to test whether MotionJudge’s preferences align with judgments made after viewing the motion?
4 comments
AuthorsOpenReview ↗

Response to Reviewer XnxU (Part 1/2)

We thank the reviewer for the positive assessment, and for finding the audit informative, MotionJudge's attribute-level feedback useful, and the model experiments thorough. We address the two comments below, with new experiments for both.


W1/Q1: The benefit of multiple LLM calls is not isolated. Compare with a single-call baseline that uses the same rubric, action schema and deterministic comparison, clarify the thinking settings, and report accuracy, latency and token usage.

Thanks for this helpful suggestion, which isolates where MotionJudge's gains come from. We ran exactly this baseline, and the separate calls do matter: with the same rubric, schema and deterministic comparison, a single call misses more errors, flags more correct captions, and is slower than MotionJudge. On the thinking settings: all judges in the paper, MotionJudge and the whole-caption judges alike, run Qwen3.5-27B with thinking disabled, temperature 0 and short answer budgets. We will state this in Section 4.

We compare four judges that add the paper's own components one at a time. MotionFact contributes the rubric, a fixed set of attributes with a severity scale, and the typed action schema, which writes a caption as an ordered list of actions whose slots (such as side, direction, speed and count) take values from closed lists. MotionJudge adds deterministic comparison, in which code rather than the LLM compares each value with the reference, and the decomposition: separate calls align the actions, read each attribute from the matched phrase alone, and quote the phrase that covers each reference action, which code checks verbatim. In paper Table 1 the schema entered together with the decomposition (“+Schema” there is MotionJudge). Here we separate them:

  • One verdict: one call that flags whether the caption contains an error.
  • +Rubric: one call with MotionFact's rubric that names the attributes it finds wrong.
  • +Schema (1 call): MotionFact's rubric and typed schema in one call. The LLM aligns the actions and fills every slot in one JSON answer, and the same code as MotionJudge compares the values with the reference. This is the baseline the reviewer suggested.
  • MotionJudge: the same schema and comparison code, plus the decomposition into separate calls.

(i) Only MotionJudge passes every attribute, and it rarely flags correct captions. The results are shown below. Each component lowers the number of attributes below threshold, and the rubric recovers the pass rate. The single-call schema lifts most single-word attributes above threshold, more in Chinese than in English, but direction and bilateral mode remain below threshold in both languages. +Schema (1 call) also reports an error on more than a quarter of correct captions, even on exact copies of the reference: when one call fills every slot at once, it reads the same words differently in the caption and in the reference, and the code then sees a mismatch that is not there.

Table A. Single-call baselines vs. MotionJudge on MotionADV (Qwen3.5-27B, thinking off). Detection: share of captions with an injected error that the judge flags. Pass: share of correct captions it leaves unflagged. Attribute rows: detection, Chinese / English. ✗: below the paper's thresholds (detection 0.90, pass 0.89). One verdict and +Rubric were re-run in the same session as +Schema (1 call) and are within 0.02 of paper Table 1.

AttributeOne verdict+Rubric+Schema (1 call)MotionJudge
side0.780✗ / 0.760✗0.876✗ / 0.894✗0.954 / 0.9450.992 / 0.982
direction0.726✗ / 0.715✗0.561✗ / 0.640✗0.820✗ / 0.765✗0.923 / 0.939
motion type0.768✗ / 0.717✗0.768✗ / 0.667✗0.926 / 0.831✗0.950 / 0.928
bilateral mode0.744✗ / 0.676✗0.720✗ / 0.648✗0.832✗ / 0.761✗0.936 / 0.901
amplitude0.477✗ / 0.467✗0.885✗ / 0.760✗0.915 / 0.838✗0.979 / 0.974
speed0.505✗ / 0.482✗0.975 / 0.9740.948 / 0.9070.974 / 0.989
oscillation axis0.588✗ / 0.535✗0.330✗ / 0.352✗0.814✗ / 0.9510.969 / 0.951
rotation0.779✗ / 0.857✗0.853✗ / 0.9520.982 / 1.000.965 / 0.966
articulation0.854✗ / 0.871✗0.891✗ / 0.877✗0.943 / 0.897✗0.948 / 0.929
count0.918 / 0.9041.00 / 1.001.00 / 1.001.00 / 0.980
lowest pass rate0.269✗ / 0.294✗0.943 / 0.9510.894 / 0.880✗0.934 / 0.894
attributes below threshold10 / 108 / 73 / 50 / 0
correct captions not flagged*1.001.000.724✗0.976

*Captions with no error (Chinese MotionADV): 281 rewordings of the reference that keep every fact, and 281 word-for-word copies of the reference. A judge should flag none of them, so 1.00 is ideal. One verdict and +Rubric reach 1.00 mainly because they flag few captions at all, as their low detection above shows.

AuthorsOpenReview ↗

Response to Reviewer XnxU (Part 2/2)

Response to Reviewer XnxU (Part 2/2)

(ii) Enabling thinking narrows the gap but does not close it, at many times the generated tokens. With thinking enabled and a 4k-token budget, +Schema (1 call) runs out of budget before answering on most samples. We therefore ran +Rubric with a 12k budget: it answers and improves, but four attributes (direction, motion type, bilateral mode, oscillation axis) stay below threshold. MotionJudge passes all four with thinking off and generates about one ninetieth of the tokens. Table B. Thinking on vs. off, Chinese MotionADV, Qwen3.5-27B. Rows: detection on the four attributes +Rubric leaves below threshold with thinking on. 12k: thinking budget. ✗: below threshold.

+Rubric, thinking off+Rubric, thinking on (12k)MotionJudge, thinking off
direction0.561✗0.759✗0.923
motion type0.768✗0.855✗0.950
bilateral mode0.720✗0.736✗0.936
oscillation axis0.330✗0.412✗0.969
attributes below threshold8 of 104 of 100 of 10
generated tokens per sample2≈4,500≈50

+Schema (1 call) with thinking on (4k budget): no answer on 1,492 of 1,633 samples.

(iii) MotionJudge takes less time per sample than the single-call baseline. On the same 100 samples and GPU, MotionJudge's calls each generate only a few tokens, while +Schema (1 call) generates one long answer. The whole-caption judges are faster still, at the accuracy cost shown in Table A.

Table C. Latency and tokens on the same GPU (100 MotionADV samples, Qwen3.5-27B).

JudgeCallsPrompt tokensGenerated tokensSeconds / sample
One verdict10012,4332000.08
+Rubric10022,6332000.08
+Schema (1 call)100124,70713,3562.75
MotionJudge2,507417,0915,0141.85

We will add the single-call baseline, the thinking settings and Tables A–C to the revision.


W2/Q2: MotionJudge has not been validated against human judgments of generated captions. Test whether its preferences agree with people who view the motion, particularly where it and the whole-caption judge disagree.

Thanks for this suggestion. We agree that a check by people who watch the motion is an important test for a metric. We ran it on generated captions, and where the two judges disagree, professional motion artists side with MotionJudge far more often than with the whole-caption judge. On HumanML3D, MotionJudge and the whole-caption LLM judge of the paper (one score of 0–2 per caption) rank the captioning models in nearly the same order (paper Table 8), but on individual motions they sometimes prefer different captions. We sampled 30 HumanML3D test motions where the two judges prefer different captions of Motion-VLM and MG-MotionLLM: in 15, MotionJudge favors Motion-VLM's caption more than the whole-caption judge does, and in the other 15 the whole-caption judge favors it more (seed fixed before rating). Three professional motion artists watched each motion, discussed it, and agreed on the better caption or “about the same”. The results are shown below.

(i) The artists agree with MotionJudge on 20 of 30 motions and with the whole-caption judge on 4.

Table D. On 30 motions where the two judges prefer different captions, whose preference do the artists agree with? Each answer is one of Motion-VLM better, MG-MotionLLM better, or about the same. The two judges give different answers, so the artists match one judge or choose the remaining answer.

Whose preference matches the artists' choiceMotions
MotionJudge20 (66.7%)
Whole-caption judge4 (13.3%)
Neither: the artists preferred Motion-VLM, which neither judge preferred5 (16.7%)
Neither: the artists preferred MG-MotionLLM, which neither judge preferred1 (3.3%)

(ii) On production captions, MotionJudge also catches more of the artists' corrections. The paper also compares the judges with professionals on 494 drafts of a production captioner that professional motion artists corrected after viewing each clip (paper Section 4.3). MotionJudge flags more of the side and direction rewrites, while the whole-caption judge marks fewer unchanged directions as wrong.

Table E. Draft–correction pairs (494 Chinese clips). Flagged: the attribute is not fully supported (MotionJudge) or named as wrong (+Rubric). MotionJudge from paper Table 24, +Rubric newly scored.

JudgeSide rewritten: flaggedSide unchanged: not marked wrongDirection rewritten: flaggedDirection unchanged: not marked wrong
Whole-caption judge (+Rubric)0.8520.9410.7650.979
MotionJudge1.000.9830.9080.897

We will add the human check (Table D) and the +Rubric row of Table E to the revision.

Thank you for the additional experiments. The single-call comparison and human evaluation address my concerns. I am keeping my original rating of weak acceptance.

Thank you for the follow-up and for your support of this work. We are glad that the single-call comparison and the human evaluation address your concerns, and we will add both to the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for all reviewers

Summary for all reviewers

We thank all reviewers for their time and constructive suggestions. We believe the main concerns have been addressed through new experiments, new analyses and clarifications, which will be incorporated into the revision. As the responses are spread across three threads, we map them here as an overview for reference. We start with the mirror test, which connects the three parts, and group the rest by component.

The mirror test: MotionADV on the motion side (Naz9, EwLW).

Instead of editing the caption, we mirror the motion. A caption counts as right only if it names the correct side on both a HumanML3D motion and its mirror, with no LLM judge. Motion-VLM is the best of all compared models, while the R-Precision evaluator cannot tell the two motions apart.

MotionJudge: checked against people and across models.

  • Human anchor (XnxU, EwLW). MotionJudge flags nearly every action, side and direction that professional motion artists rewrote in 494 production drafts. On 30 motions where it and the whole-caption judge prefer different generated captions, three artists side with MotionJudge on 20 and with the whole-caption judge on 4.
  • The separate calls matter (XnxU, Naz9). With the same rubric, schema and comparison code in one call, the judge leaves attributes below threshold and flags correct captions, even with thinking at many times the tokens. MotionJudge passes every attribute and is faster per sample.
  • A frontier model in one call is not enough (Naz9, EwLW). Claude Opus 5.5 finds wrong values well, and with MotionFact's schema it also catches omitted facts, but in one call it still flags correct captions. MotionJudge passes every attribute with an open 27B model, offline and reproducible.
  • Not tied to one model family (EwLW). With GLM-4-32B or Llama-3.3-70B as the judge model, MotionJudge leaves far fewer attributes below threshold than single-call judging. Under gemma-4-31B, the ranking of the captioning models is unchanged.
  • Real errors (XnxU, EwLW). On the same drafts, MotionJudge catches more side and direction corrections than the whole-caption judge, and marks fewer unchanged sides as wrong.

Motion-VLM: evidence beyond our judges.

  • More facts, stated as accurately (EwLW). Motion-VLM states more reference facts correctly than every baseline, and on the facts both captions state it is at least as precise as the strongest baselines.
  • What the full model adds (Naz9, EwLW). Over summary-only, the structured target and the motion tokens both help, the tokens most on Motion-X++ hand clips.
  • Skeleton renders are enough to learn from (Naz9). On Motion-X++, Motion-VLM trained on the fast skeleton renders scores far above the best motion LLM, which even the zero-shot Qwen3-VL-32B already exceeds.
  • General ability (Naz9). Trained on motion data alone, Motion-VLM largely keeps the general image and video ability of its base model, and even improves on MMStar.

The audit and the main results (EwLW).

  • R-Precision rewards wrong captions. The human references score below both models on the official evaluator, and on test motions listed by ID it prefers captions with a wrong side, speed or hand position.
  • Blind metrics change real comparisons. Among close models, BLEU-4 reverses an order that MotionJudge separates. Where MotionJudge has a clear preference, ROUGE-L reverses it on about one motion in three, and captions with every side made wrong still rank above both baselines on ROUGE-L and CIDEr.
  • The fact-dense subset. On fact-dense references, three quarters of the test set, Motion-VLM leads both baselines significantly under MotionJudge, while ROUGE-L gives the same small margin at every fact count.

Contributions and release.

  • Contributions (Naz9). MotionFact builds captions from meta-actions with orthogonal slots, MotionJudge is an evaluation protocol (harness) rather than a prompt, and Motion-VLM shows that a general VLM needs no dedicated motion tokenizer to caption motion well.
  • Release (EwLW). Upon acceptance, we will open-source the 494 clips: the production drafts, the artists' corrected captions, the motion data and the renders.

We apologize for posting our responses late in the discussion period. We wanted to answer every concern in full, with new experiments, in a single round. We hope these responses address the reviewers' concerns, and we would be grateful if they could be considered in the final ratings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.