acceptodds
Withdrawn from ICLR 2027

HumanMoveVQA: Can VideoMLLMs reason about Human Movement from Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

23 views

Abstract

Multimodal Large Language Models (MLLMs) can describe what a person does in a video, but not where they go. Answering “does the person end up left of where they started?” or “how many times do they turn clockwise?” requires reasoning about global trajectory and orientation in 3D over time, beyond apparent motion in the image, which existing benchmarks, focused on scene-level events or local joint articulation, do not probe. We introduce HumanMoveVQA, a benchmark for reasoning about global human trajectory and orientation in exocentric video. We propose a scalable multi-stage pipeline that lifts 2D videos from five human-motion datasets into world-consistent 3D SMPL-X motion tracks anchored to the person’s first-frame position and heading, and converts them into discrete motion events. From these, we generate 12,282 multiple-choice questions over 127 test videos (15–90 s) across seven categories spanning event detection, aggregation, ordering, and trajectory-level inference. The best of 12 Video MLLMs reaches a chance-normalised score of 14.0, against 78.7 for humans. Fine-tuning an open-source model on our world-consistent supervision roughly triples its score, showing that the problem is learnable.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Video Understanding, Human Motion Understanding, Spatial Reasoning, Video Question Answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MkBqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HumanMoveVQA to evaluate video multimodal models' understanding of global human trajectories and orientation changes. The authors reconstruct 3D trajectories from RGB videos drawn from five human-motion datasets, express motion relative to each person's initial position and heading, segment it into discrete events, and generate seven categories of multiple-choice questions. The test set contains 12,282 questions over 127 videos; models receive only the video, question, and answer options. The experiments show that current models perform poorly on event ordering and trajectory-level inference. Fine-tuning Qwen3-VL 4B on the constructed dataset improves its chance-normalized Score from 12.25 to 38.96. The paper further examines label quality and fine-tuning gains through rescoring with ground-truth-derived answers, cross-dataset evaluation, and several ablations.

Critical strengths
  1. The benchmark targets a valuable capability. Tracking a person's displacement, turns, and final position over a relatively long video requires sustained integration of spatial and temporal information beyond local action recognition. The paper makes this capability measurable through a fixed initial reference frame and seven question categories, covering moving cameras, multiple viewpoints, and multi-person scenes. Computing answers from 3D trajectories also provides traceable geometric evidence for the labels, facilitating analysis of specific model errors.

  2. The dataset's value for training is supported by substantial validation. With the same backbone and 32-frame input, Qwen3-VL 4B's Score improves from 12.25 to 38.96. The clip-level bootstrap accounts for correlations among questions from the same video, and the fine-tuned model remains substantially better than the base model when rescored using answers derived from ground-truth trajectories. Cross-dataset transfer and question-paraphrasing experiments further strengthen confidence in this result. These validations extend the contribution beyond a single evaluation of existing models.

Critical weaknesses
  1. The sensitivity of model performance in the key categories to the label source remains unclear. On comparable questions with fixed answer options, Table 12 reports average answer flip rates of approximately 10% for the four validated sources. Ordering has flip rates of approximately 13%–16%, while Trajectory Affordance reaches 20% on SLOPER4D. These results show that ordering and trajectory labels are more sensitive to reconstruction errors than Existence labels. Table 13 already establishes that the overall fine-tuning advantage persists, but reports only the Score averaged over four sources, without the models' category-level accuracies under ground-truth-derived answers. It therefore remains unclear how much the conclusions about category difficulty and category-specific fine-tuning gains depend on the label source. For a benchmark paper, these results would help establish which model comparisons the benchmark can reliably support.

  2. The ordering results still lack an analysis of the fine-tuned model's dependence on visual input. In Table 8, text-only Gemini achieves 35.61% on EgoBody's Ordering category, compared with 35.48% for fine-tuned Qwen3-VL 4B. This cross-model comparison suggests that some questions are answerable from text, but does not establish whether Qwen uses the video. The authors discuss residual language bias, yet the submission does not report a no-video control for the fine-tuned Qwen itself. The paraphrasing experiment tests robustness to wording changes, but does not rule out reliance on statistical regularities in the questions and options. The authors also discuss sparse frame sampling as a possible limitation and compare frame counts in Table 17; however, they report only aggregate Scores per source and change the frame count jointly during training and testing. The results support weak ordering performance under the evaluated settings. Distinguishing limitations in visual observation from difficulties in reasoning about event relationships requires corresponding controls and category-level results.

Questions for authors
  1. How do the category-level results for Ordering and Trajectory Affordance change when using the ground-truth-derived answers in Table 13? Please report the base and fine-tuned models' results on the same comparable questions to establish whether the conclusions about task difficulty and fine-tuning gains hold for these two categories.
  2. Could you provide a no-video control for the fine-tuned Qwen, particularly on Ordering? What are the Ordering accuracies under the 16-, 32-, and 64-frame settings in Table 17? These results would help assess the effects of language bias and temporal sampling on ordering performance.
  3. Equation (1) expresses translation only as p_t − p_0, while the text states that the X/Z axes are aligned with the person's initial heading. Have the positions already been rotated in an earlier step, or does the equation omit the inverse initial-yaw rotation? Please clarify the coordinate transformation used in the implementation and how it is applied during label generation.
Reviewer_5cHfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for reasoning about global human trajectory and orientation. The 2D videos are reconstructed into 3D trajectories which are then converted to discretized spatial codes. Identity descriptions based on appearance are added, unreliable motion is removed and finally, questions are generated. The questions cover 8 categories, mostly related to the generated trajectory and are multiple choice QA with the dataset derived from existing human motion videos. Several video LLMs are shown to perform poorly on the benchmark when compared to humans. It is further shown that fine-tuning open source models triples the original scores.

Critical strengths
  1. The paper identifies a meaningful gap in video understanding. Understanding a person’s global trajectory and orientation is a useful capability that activity recognition and joint-level motion descriptions do not fully capture. The motivation is particularly convincing when the camera moves, because apparent image motion can differ substantially from actual person motion.
  2. The data-generation pipeline produces traceable answers. Answers are computed from structured motion events rather than freely generated by an LLM. This makes it possible to inspect the geometric quantities, event boundaries, and rules supporting an answer. The benchmark includes varied recording conditions. The five sources cover static and moving cameras, different trajectory lengths, indoor and outdoor environments, and multiple-person scenes.
Critical weaknesses
  1. The experiments do not separate failures in visual perception, motion estimation, and reasoning. A wrong answer can result from several different problems such as tracking the wrong person, incorrectly estimating movement, failing to separate camera and person motion, or correctly observing events but aggregating, ordering or reasoning them incorrectly. The benchmark measures the combined outcome of all these stages. Consequently, low accuracy does not reveal which capability failed. Since there are many intermediate steps between the input video and the final answer, e.g. pose reconstruction, root trajectory generation etc., the accuracy becomes a function of several of these intermediate steps. The discretization threshold may cause a small change in, say, the reconstruction error to change the label. The authors do mention this in the limitations and address it in detail in the appendix, but the fundamental question is whether the accuracy of the framework is the actual accuracy of motion understanding or is it reflecting the accuracy of the reconstruction pipeline.
  2. Some of the claims are over stated. It seems the coordinate frame is defined relative to the subject in the first frame. In that case, it may not be appropriate to call it 'world coordinates' since the coordinate frame moves with the subject. It may be necessary to define the relationship between, say, translation and orientation explicitly, e.g., the right in 'move right' could mean right in the image or camera coordinates or relative to the heading etc. Questions related to trajectory affordance could be considered more on spatio-temporal trajectory understanding than 'reasoning'. The conclusion reached regarding comparison with MotionLLMs, e.g .line 374, is that joint level motion supervision does not work for global trajectory reasoning. This could be a reason but the experiment does not establish it fully. A better way could be to train a base architecture in 3 different ways - joint level supervision, trajectory level supervision and both. Then evaluate on both benchmarks. It is interesting to note that HumanMoveVQA degrades performance of ActionArt, but improves Qwen3-VL, showing that both local and global motion representations are useful.
Questions for authors
  1. How well does the fine-tuned Qwen3-VL perform without video input? The paper reports a text-only baseline only for Gemini, whereas most subsequent analyses use Qwen3-VL. A corresponding text-only evaluation of the fine-tuned Qwen3-VL would help establish how much its performance depends on visual evidence rather than regularities in the questions and answer choices.
  2. Could the authors conduct a diagnostic input ablation comparing performance on the same questions using (i) RGB video, (ii) textual motion-event descriptions derived from reconstructed tracks, and (iii) equivalent descriptions derived from ground-truth tracks for some of the evaluated models? This could help in assessing the first weakness.
  3. The experiment mentioned in the last part of the second weakness could establish the weakness of joint level supervision.
Reviewer_jtyZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HumanMoveVQA, a benchmark for evaluating Video MLLMs' understanding of global human motion, including displacement, orientation, and temporal ordering. The authors propose an automated pipeline that reconstructs 3D human trajectories from videos and generates multiple-choice questions across seven categories. The benchmark contains 12,282 questions from 127 test videos.

Experiments with 12 Video MLLMs show a substantial gap between current models and human performance. Fine-tuning on the proposed dataset significantly improves performance, although temporal ordering remains challenging.

Critical strengths
  1. Meaningful evaluation gap. The paper highlights a relatively underexplored aspect of video understanding: reasoning about global human trajectories and orientation rather than local body movements or action recognition. The viewpoint-consistent formulation is particularly interesting.

  2. Scalable benchmark construction and extensive evaluation. The automated pipeline enables large-scale QA generation from existing motion datasets. The authors evaluate a broad range of MLLMs and provide useful analyses of fine-tuning, cross-dataset generalization, and viewpoint sensitivity.

Critical weaknesses
  1. Insufficient motivation for evaluating these tasks with MLLMs. The paper motivates the benchmark through applications such as robotics, navigation, and industrial safety. However, most questions can be answered deterministically once 3D trajectories are extracted, as demonstrated by the authors' own data generation pipeline. It remains unclear why general-purpose Video MLLMs need to perform these tasks directly, or what advantages they offer over specialized motion tracking and geometric computation. While evaluating these capabilities in MLLMs is not inherently unreasonable, the paper does not sufficiently establish their importance for general-purpose video understanding.

  2. Evaluation validity and unclear sources of model failure. The benchmark relies on automatically reconstructed 3D trajectories, and the authors acknowledge that approximately 10% of answers change when using ground-truth motion tracks (Section A.2). This raises concerns about label reliability, particularly for temporal ordering and trajectory-level questions. Moreover, the evaluation does not clearly separate failures in recovering 3D motion from failures in reasoning over that motion. As a result, it is difficult to determine whether the reported low performance reflects limitations in spatial-temporal reasoning or simply inaccurate motion perception.

Questions for authors
  1. Could the authors clarify why global trajectory reasoning should be performed directly by general-purpose Video MLLMs, rather than through specialized motion tracking and geometric computation? What practical capabilities would strong performance on this benchmark demonstrate beyond those provided by existing motion analysis pipelines?

  2. Have the authors considered evaluating a structured motion-based baseline or providing ground-truth motion representations to an LLM? Such experiments could help distinguish failures in motion perception from failures in reasoning.

  3. Given that approximately 10% of answers change when using ground-truth trajectories, how does this label noise affect the reliability of model comparisons, especially for Ordering and Trajectory Affordance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.