- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether weak fine-grained motion-caption scores come from the models or from metrics that cannot see a single wrong fact. It builds MotionADV, minimal caption pairs differing in exactly one fact from 494 artist-corrected clips, and shows that n-gram and embedding metrics barely react to the error, and that n-gram metrics even prefer the wrong caption to a fact-preserving paraphrase. It then contributes MotionFact, an eleven-attribute rubric over a typed action schema, MotionJudge, a decomposed protocol that checks each attribute against a reference and, with Qwen3.5-27B, reaches at least 0.90 detection and 0.89 pass on every attribute, and Motion-VLM, a Qwen3.5-4B fine-tune on multi-view renders, motion tokens, and fact-first targets that outperforms five published captioners on HumanML3D.
Critical strengths
1. The audit quantifies a blind spot in motion-caption metrics On minimal pairs built from 494 artist-corrected clips, a single wrong fact moves every similarity metric less than a fifth of the way toward an unrelated caption, and n-gram metrics rank the wrong caption above a fact-preserving paraphrase in at least 1,258 of 1,261 Chinese pairs. This matters because standard scores cannot certify the physical facts of a caption, which is the gap fact-level evaluation must close.
2. The results come with the controls and records needed to check them The paper reports ceiling, floor, and repeat-noise controls, a label-leakage subset excluding the judge's base model, a replication under a second base model, an explicit accounting of its own misses on real errors, and the release of MotionADV with per-sample validator votes. This matters because the challenge set and its analyses can be checked and reused step by step, making them a durable resource for fact-level evaluation.
Critical weaknesses
1. MotionJudge lacks a human anchor and its validity chain runs through one model family The attribute edits of MotionADV are generated by Qwen3.5-27B, the labels are validated by three language models with escalated cases decided by GPT-5.5, and MotionJudge is built on Qwen3.5-27B, so one family generates the edits, contributes two of the three label validators, and supplies the judge base. The paper itself raises the risk that the judge shares blind spots with the labels, and the labels are filtered by models rather than humans. Two mitigations are reported. Detection stays at or above 0.94 on the subset whose labels pass without the Qwen vote, and gemma as a second base model reproduces the ranking, but several attributes fall below the paper's own thresholds of 0.90 detection and 0.89 pass, with bilateral-mode detection at 0.888 and 0.831 and motion-type pass at 0.833 and 0.788. On real errors the paper reports contradicted rates of only 30% and 20% for side and direction rewrites and lists 22 misses, 13 caused by alignment rejecting a matching action and the rest by a post-rule reassignment, a value read that misses the side, or a reference parse with no usable fact. The grader assigns each miss a score of 2, but the attribute-level verdicts remain unreliable on real errors, and that verdict level is the stated advantage over a whole-caption judge. MotionJudge has not been validated against human raters, so the replacement for the metrics the paper calls unreliable has no validity evidence independent of models, and both the audit and the model conclusions rest on an instrument validated by models alone.
2. The model-side claim rests on self-built judges, and the fact-level evidence shows coverage rather than accuracy The advantage of Motion-VLM over the strongest published baseline, MG-MotionLLM, is only +0.057 on MotionJudge (2.483 against 2.421) and +0.033 on the whole-caption judge, both of which are built in this paper and share the Qwen family with Motion-VLM, and the gemma replication keeps the ranking but does not meet the same reliability thresholds. The authors' own summary-only baseline already matches MG-MotionLLM, so the structured target and the motion tokens contribute only +0.061 and +0.012 over plain fine-tuning. The error rate among stated facts is 12.4% against 12.5% for that baseline, and the +1.5 point gain in supported facts comes mainly with fewer omissions (67.5% to 66.2%) while the contradiction rate barely moves (+0.2), so the model states more facts rather than more accurate ones. On the established R-Precision@1 protocol outside the two self-built judges, MG-MotionLLM still ranks first, and the paper explains this through an encoder-insensitivity finding from its own audit. Because the fact-level precision gain attributed to the model input and training target does not appear in the paper's own comparison, the accuracy claim rests only on in-house instruments, and the second contribution remains unsubstantiated at the mechanism level.
3. The audit is disconnected from the main experiments and its premise is unproven on the main benchmark The paper attributes low scores on motion facts to both weak models and insensitive metrics, but its main model comparisons fall where the two do not disagree. On HumanML3D, BLEU-4 ranks the nine systems exactly as MotionJudge does (Spearman 1.00), ROUGE-L agrees at 0.98, and the single-call whole-caption judge also agrees at 0.98, so the system ranking in the main table can be reproduced with existing metrics and the need for a protocol costing about 17 calls per caption is never demonstrated. The paper explains the agreement by noting that HumanML3D references are short and state few facts, which concedes that the main benchmark cannot exercise the problem the paper sets out to fix. The divergence that changes a conclusion appears in the zero-shot comparison on Motion-X++, and the paper does not quantify how often or how severely metric blindness affects real system comparisons, so the case for replacing the metrics still rests mainly on synthetic probes and the empirical basis for the metric side of the thesis remains weak.
Questions for authors
Has MotionJudge been compared against even a few hundred human adjudications, and how does human agreement vary across attributes? On real errors, how much of the advantage of attribute-level verdicts over the whole-caption judge remains, and do the 30% and 20% contradicted rates on side and direction rewrites change the conclusion that decomposition outperforms whole-caption scoring? Do the attributes that fall below the thresholds under the gemma base change the system ranking of the main table, or only the absolute scores? On HumanML3D or any fact-dense subset, is there any instance where MotionJudge and BLEU-4 order the systems differently?
Please add a small human validation study and report agreement rates, and recalibrate the detection and pass thresholds against it. Re-report the per-attribute results on real errors and under the second base model, and provide a system-level comparison on a fact-dense subset to connect the audit findings to the model results. Split the model claim into separately testable coverage and precision statements, or add conditional precision evidence. Clarify the licensing and release plan for the 494 industrial clips, and state the minimal essential difference from KPM-Bench and MotionHalluc in the main text.
7 comments
Response to Reviewer EwLW (Part 2/6 - W1 & Q2, Q3)
W1 - (iii) & Q3: Do the attributes below threshold under gemma change the ranking, or only the absolute scores? (Q3)
They change only the absolute scores. Under gemma-4-31B, Motion-VLM remains first, and its margin over MG-MotionLLM grows (Table D). Gemma flags motion-type errors more often than Qwen, including on some correct captions. This over-flagging lowers every model's score, but not their order. Table D lists the MotionJudge scores, from 0 to 5, under each judge model.
Table D. Ranking under two judge models (HumanML3D).
| Qwen3.5-27B | gemma-4-31B | |
|---|---|---|
| MotionGPT | 2.303 | 2.047 |
| Motion-Agent | 2.375 | 2.132 |
| MG-MotionLLM | 2.421 | 2.154 |
| summary-only | 2.420 | 2.210 |
| Motion-VLM | 2.483 | 2.253 |
| Motion-VLM − MG-MotionLLM | +0.057 [+0.019, +0.096] | +0.100 [+0.058, +0.141] |
| motion type, detection / pass, Chinese | 0.950 / 0.934 | 0.983 / 0.833 |
| motion type, detection / pass, English | 0.928 / 0.934 | 0.952 / 0.788 |
W1 - (iv) & Q2: On real errors, do the low contradicted rates for side and direction change the conclusion that decomposition helps? (Q2)
Table E compares MotionJudge with the whole-caption judge on the sides and directions that the artists rewrote or left unchanged. A value counts as flagged when the judge marks it as wrong or not fully supported.
- Many rewrites change the whole action, where “not mentioned” is correct. About a third of the artists' side rewrites and more than half of their direction rewrites replace, add or delete the whole action. Of the rewrites that MotionJudge either marks as contradicted or misses, it marks most as contradicted, and even a miss lowers the caption's overall score to 2 of 5.
- The attribute-level advantage remains on real errors. We scored the same 494 pairs with the whole-caption judge that names the wrong attributes. MotionJudge flags more of the artists' side and direction rewrites and marks fewer unchanged sides as wrong. The whole-caption judge marks fewer unchanged directions as wrong, but misses a quarter of the direction rewrites.
- Most misses come from one alignment step, and a simple retry reduces them. They happen because the step that matches caption actions to reference actions refuses to match two actions that differ only in side or direction, so a wrong side is recorded as “not mentioned” instead of wrong. Retrying the match once with the side and direction words hidden (align*) raises the share of rewrites marked as a wrong value from 30% to 41% for side and from 20% to 29% for direction, without changing the other rates. We will adopt it in the revision.
Table E. MotionJudge vs. the whole-caption judge on real errors (494 drafts).
| Judge | Side rewritten: flagged | Side unchanged: not marked wrong | Direction rewritten: flagged | Direction unchanged: not marked wrong |
|---|---|---|---|---|
| Whole-caption judge (+Rubric) | 0.852 | 0.941 | 0.765 | 0.979 |
| MotionJudge | 1.00 | 0.983 | 0.908 | 0.897 |
We will add Tables A–E and the alignment retry to the revision.
Response to Reviewer EwLW (Part 3/6 - W2)
W2: The model-side claim rests on self-built judges, and the fact-level evidence shows coverage rather than accuracy.
Thanks for this suggestion. In short, Motion-VLM's advantage holds without our own judges, and it states more facts at no loss of precision. Following the suggestion, we add a judge-free mirror test and a judge from another family, and we split the claim into coverage and precision. We answer four questions:
- (i) Does the margin hold without our judges? Yes. Motion-VLM is first on a mirror test that uses no LLM judge (Table F), and also with gemma as the judge.
- (ii) Does summary-only already match MG-MotionLLM? Yes, because it already uses our three-view input. Our full model is still significantly better than it.
- (iii) Is the gain coverage or precision? Both. Motion-VLM states more correct facts than every baseline, and on the facts that both models state, it makes no more errors. The extra facts it states are as accurate as the rest (Table H).
- (iv) Why does MG-MotionLLM lead on R-Precision@1? Because R-Precision does not check facts. It prefers captions with a wrong side, speed or position, it cannot tell left from right, and it even ranks the human references below both models (Tables I and J).
The details follow.
--
W2 - (i) Does the margin over MG-MotionLLM hold without our judges?
- A mirror test needs no LLM judge (Table F). HumanML3D pairs every test motion with its exact mirror, whose reference swaps left and right. A caption counts as right only if it names the correct side on both. Motion-VLM leads every model, MG-MotionLLM included. Table F gives the share of pairs right on both sides and the difference from Motion-VLM with its 95% interval, which excludes zero for every model.
- Other metrics and a judge from another family agree. Motion-VLM also leads the n-gram and embedding metrics on HumanML3D (paper Table 2), and it ranks first with gemma as the judge model (Table D).
Table F. Mirror test, no LLM judge (HumanML3D, 660 pairs).
| Model | Side right on motion and mirror | Motion-VLM − model |
|---|---|---|
| DEMO | 0.118 | +0.291 [+0.253, +0.329] |
| MotionGPT3 | 0.189 | +0.220 [+0.182, +0.259] |
| MotionGPT | 0.261 | +0.148 [+0.109, +0.188] |
| Motion-Agent | 0.277 | +0.132 [+0.097, +0.165] |
| MG-MotionLLM | 0.326 | +0.083 [+0.049, +0.120] |
| Motion-VLM | 0.409 | – |
W2 - (ii) Does the summary-only baseline already match MG-MotionLLM?
It does, but it is not plain fine-tuning. Summary-only already uses our three-view, color-coded input, one of our contributions and the largest single gain in our ablations. The full model is still significantly better than summary-only, with either judge model and on the judge-free mirror test (Table G). Both added parts help. The structured target gives a significant gain on HumanML3D. The motion tokens give a large gain on Motion-X++, whose motion data includes hand and finger joints, about twice as large on hand clips, whose reference mentions hands or fingers, as on the others. In Table G, significant means that the 95% interval of the paired difference excludes zero. They add little on HumanML3D, where the render already shows every joint.
Table G. What the full model adds over summary-only.
| Change | Benchmark | Measure | Without → with | Significant |
|---|---|---|---|---|
| summary-only → full model | HumanML3D | MotionJudge (Qwen) | 2.420 → 2.483 | yes |
| summary-only → full model | HumanML3D | MotionJudge (gemma) | 2.210 → 2.253 | yes |
| summary-only → full model | HumanML3D | mirror test | 0.351 → 0.409 | yes |
| adding the structured target | HumanML3D | MotionJudge | 2.420 → 2.465 | yes |
| adding the motion tokens | Motion-X++ | MotionJudge | 1.584 → 1.805 | yes |
| ↳ on hand clips | Motion-X++ | MotionJudge | 1.728 → 2.028 | yes |
| ↳ on other clips | Motion-X++ | MotionJudge | 1.388 → 1.528 | yes |
Response to Reviewer EwLW (Part 1/6 - W1 & Q1)
We thank the reviewer for the careful and detailed review, and for recognizing that the audit quantifies a blind spot that standard scores cannot certify and that the paper reports the controls needed to check it. We address the three weaknesses, the four questions and the requests below, with new experiments for each.
W1: MotionJudge lacks a human anchor, and its validity chain runs through one model family.
Thanks for this careful analysis and the concrete suggestions. In short, MotionJudge agrees with human judgments on real errors and on generated captions, and it does not depend on one model family. Following the suggestions, we report its agreement with the artists' corrections of 494 production drafts per attribute, and we further add a human check on generated captions. We split the concern into four questions:
The details follow.
W1 - (i) & Q1: Has MotionJudge been compared against human adjudications, and how does agreement vary across attributes?
Table A. MotionJudge vs. the artists' corrections (494 drafts, Chinese).
Table B. Artists' choice where the two judges disagree (30 HumanML3D motions).