- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HumanMoveVQA to evaluate video multimodal models' understanding of global human trajectories and orientation changes. The authors reconstruct 3D trajectories from RGB videos drawn from five human-motion datasets, express motion relative to each person's initial position and heading, segment it into discrete events, and generate seven categories of multiple-choice questions. The test set contains 12,282 questions over 127 videos; models receive only the video, question, and answer options. The experiments show that current models perform poorly on event ordering and trajectory-level inference. Fine-tuning Qwen3-VL 4B on the constructed dataset improves its chance-normalized Score from 12.25 to 38.96. The paper further examines label quality and fine-tuning gains through rescoring with ground-truth-derived answers, cross-dataset evaluation, and several ablations.
Critical strengths
-
The benchmark targets a valuable capability. Tracking a person's displacement, turns, and final position over a relatively long video requires sustained integration of spatial and temporal information beyond local action recognition. The paper makes this capability measurable through a fixed initial reference frame and seven question categories, covering moving cameras, multiple viewpoints, and multi-person scenes. Computing answers from 3D trajectories also provides traceable geometric evidence for the labels, facilitating analysis of specific model errors.
-
The dataset's value for training is supported by substantial validation. With the same backbone and 32-frame input, Qwen3-VL 4B's Score improves from 12.25 to 38.96. The clip-level bootstrap accounts for correlations among questions from the same video, and the fine-tuned model remains substantially better than the base model when rescored using answers derived from ground-truth trajectories. Cross-dataset transfer and question-paraphrasing experiments further strengthen confidence in this result. These validations extend the contribution beyond a single evaluation of existing models.
Critical weaknesses
-
The sensitivity of model performance in the key categories to the label source remains unclear. On comparable questions with fixed answer options, Table 12 reports average answer flip rates of approximately 10% for the four validated sources. Ordering has flip rates of approximately 13%–16%, while Trajectory Affordance reaches 20% on SLOPER4D. These results show that ordering and trajectory labels are more sensitive to reconstruction errors than Existence labels. Table 13 already establishes that the overall fine-tuning advantage persists, but reports only the Score averaged over four sources, without the models' category-level accuracies under ground-truth-derived answers. It therefore remains unclear how much the conclusions about category difficulty and category-specific fine-tuning gains depend on the label source. For a benchmark paper, these results would help establish which model comparisons the benchmark can reliably support.
-
The ordering results still lack an analysis of the fine-tuned model's dependence on visual input. In Table 8, text-only Gemini achieves 35.61% on EgoBody's Ordering category, compared with 35.48% for fine-tuned Qwen3-VL 4B. This cross-model comparison suggests that some questions are answerable from text, but does not establish whether Qwen uses the video. The authors discuss residual language bias, yet the submission does not report a no-video control for the fine-tuned Qwen itself. The paraphrasing experiment tests robustness to wording changes, but does not rule out reliance on statistical regularities in the questions and options. The authors also discuss sparse frame sampling as a possible limitation and compare frame counts in Table 17; however, they report only aggregate Scores per source and change the frame count jointly during training and testing. The results support weak ordering performance under the evaluated settings. Distinguishing limitations in visual observation from difficulties in reasoning about event relationships requires corresponding controls and category-level results.
Questions for authors
- How do the category-level results for Ordering and Trajectory Affordance change when using the ground-truth-derived answers in Table 13? Please report the base and fine-tuned models' results on the same comparable questions to establish whether the conclusions about task difficulty and fine-tuning gains hold for these two categories.
- Could you provide a no-video control for the fine-tuned Qwen, particularly on Ordering? What are the Ordering accuracies under the 16-, 32-, and 64-frame settings in Table 17? These results would help assess the effects of language bias and temporal sampling on ordering performance.
- Equation (1) expresses translation only as
p_t − p_0, while the text states that the X/Z axes are aligned with the person's initial heading. Have the positions already been rotated in an earlier step, or does the equation omit the inverse initial-yaw rotation? Please clarify the coordinate transformation used in the implementation and how it is applied during label generation.