- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MMSI-Video-Bench, a human-annotated benchmark containing 1,106 multiple-choice questions grounded in 1,278 video clips from 25 public datasets and newly recorded videos. It covers spatial construction, motion understanding, planning, prediction, and cross-video reasoning, with questions and explanatory rationales developed and reviewed by 3D vision researchers. The authors evaluate 25 open-source and proprietary models under two frame-input settings, finding low overall accuracy and a substantial gap from the reported human baseline. An analysis of 520 incorrect responses further characterizes failures in visual grounding, geometric reasoning, instruction interpretation, and cross-video inference.
Critical strengths
- The benchmark combines diverse video sources with expert-written questions, explanatory rationales, and cross-review, providing a valuable alternative to narrowly scoped benchmarks built through automatic templates.
- Its coverage of spatial, temporal, and cross-video tasks enables a broader assessment of current models, while subtype-level results reveal weaknesses that aggregate video-understanding scores can obscure.
Critical weaknesses
- The paper does not establish how strongly the questions depend on the spatial and temporal evidence they are intended to test. For example, the memory-update question in Figure 2 labels “Someone went to school” as the answer to an unobserved event between recordings, but the presented example does not explain why this conclusion is uniquely supported over other plausible explanations. Expert agreement alone does not resolve the distinction between a visually grounded answer and a shared commonsense assumption. The main-text experiments also lack controls that remove temporal information or one of the videos, leaving it unclear whether the proposed categories consistently require their claimed capabilities.
- Section 4.3 interprets poor performance by spatially adapted models as evidence of limited spatial generalization, even though it explicitly identifies degraded instruction following as a major source of failure. VLM3R’s accuracy of 4.97%, far below the approximately 24% random baseline, makes separating these factors particularly important. Without distinguishing invalid or noncompliant responses from valid but spatially incorrect answers, the results cannot establish whether the learned spatial capabilities themselves fail to transfer. Similarly, the error analysis infers limitations in temporal aggregation and internal representations from response-level error labels without directly testing those explanations, weakening the benchmark’s claimed diagnostic value.
Questions for authors
- What specific visual evidence supports “Someone went to school” in Figure 2 over the alternative answer choices, and how were questions about unobserved events checked to ensure that the correct answer was supported by the videos rather than primarily by commonsense assumptions?
- For Spatial-MLLM and VLM3R, what proportion of failures comes from instruction following or answer formatting, and what proportion consists of valid responses with incorrect spatial reasoning? Does the conclusion about poor spatial generalization still hold when these failure types are evaluated separately?
4 comments
Author Response to Reviewer EzaT (2/2)
Response to Weakness1 & Question 1 (Part2)
3. The need to integrate information from both videos in Cross-Video Reasoning tasks
The frame-sampling experiments in Appendix C.2 (Frame Sampling Study) demonstrate the importance of video-derived visual spatial information for model performance on MMSI-Video. During the rebuttal period, we additionally conducted additional ablation experiments to demonstrate that answering Cross-Video Reasoning questions requires integrating spatiotemporal information from both videos:
When one video was removed, all four representative models (o3, GPT-4o, Gemini-2.5-Flash, Qwen2.5-VL-7B, InternVL2.5-38B ) in the table showed substantial performance drops, with their performance approaching the level of random guessing. These results demonstrate the importance of spatiotemporal information in these tasks.
| Model | Both videos (%) | One video (%) |
|---|---|---|
| o3 | 34.9 | 29.7 (−5.2) |
| GPT-4o | 29.6 | 25.6 (−4.0) |
| Gemini-2.5-Flash | 34.3 | 31.4 (−2.9) |
| Qwen2.5-VL-7B | 26.7 | 23.3 (−3.4) |
| InternVL2.5-38B | 32.6 | 27.9 (−4.1) |
For Weakness 2 & Question 2
As mentioned in our paper, these models’ capabilities do not generalize effectively to other benchmarks, and their original abilities may even be impaired. One of the affected abilities is instruction following.
In fact, when evaluating these models, to mitigate this effect and make the evaluation as effective as possible, we used an answer-extraction method designed to be as robust as possible to extract valid answers wherever possible. As a result, all responses from SpaceQwen and nearly all responses from Spatial-MLLM were valid, so their evaluation scores can directly reflect the spatial capabilities learned and transferred by these models. In contrast, VLM3R exhibited severe degradation in instruction-following ability, with most responses consisting of repetitive numerical outputs, resulting in an abnormally low score. Its evaluation score was clearly affected by its instruction-following ability. Furthermore, to provide additional supporting evidence, we evaluated larger spatially fine-tuned models, RoboBrain 2.0 (32B) and SpatialThinker (30B). Their responses were entirely valid, so their scores can directly reflect the spatial capabilities learned and transferred by these models.
| Method | Acc. (vs. baseline) | Failure Rate |
|---|---|---|
| SpaceQwen (3B) | 27.58 (−0.09) | 0/1106 |
| Spatial-MLLM (7B) | 24.05 (−4.43) | 5/1106 |
| VLM3R (7B) | 4.97 (−23.51) | 905/1106 |
| RoboBrain 2.0 (32B) | 30.04 (+1.40) | 0/1106 |
| SpatialThinker (30B) | 30.11 (+1.31) | 0/1106 |
As can be seen, only VLM3R is affected by invalid or non-instruction-following responses. For the other models, their evaluation scores can directly reflect the spatial capabilities they have learned and transferred, with essentially no influence from invalid or non-instruction-following responses. Compared with their corresponding baselines, these models show little to no performance improvement on MMSI-Video-Bench, indicating that their capabilities do not generalize effectively to other benchmarks. After accounting for non-instruction-following responses, the conclusion of limited spatial generalization still holds.
Thank you for your response and clarification. My remaining concern is whether the Memory Update questions require temporal reasoning beyond comparing two snapshots. In Figure 2, a model could select A simply by noticing that the schoolbag is gone, without inferring what happened between the recordings. Removing one video also prevents that comparison, so the drop in accuracy leaves open whether one representative frame from each video would suffice. This distinction matters for interpreting what a correct answer tells us about the model’s capabilities.
We thank the reviewer for the follow-up and constuctive feeback. Below, we explain why one representative frame from each video is insufficient to answer this example and clarify the different levels of spatiotemporal understanding assessed by our Memory Update tasks.
- One representative frame from each is insufficient.
As stated in our rebuttal, our question-design criteria ensure non-trivial spatial and temporal reasoning: “The questions could not be resolved by referring to a single key frame or a short temporal window.” This requirement also applies to the example in Figure 2.
The schoolbag is not necessarily present in the first video (if it is absent, the option is also incorrect). Furthermore, its absence from its original location in the second video may may be because it has disappeared from the scene or has been moved to another location within the same scene (both scenarios are possible in our tasks). From a representative frame, a model can only determine that the schoolbag is no longer at its original location.It cannot determine whether the schoolbag has disappeared from the scene or remains elsewhere within it.
Answering the question therefore requires careful information retrieval, object localization, and matching across both long video sequences rather than from two representative frames. (The schoolbag might be visible in a corner of a particular frame, or it might be absent from the scene altogether) Moreover, both videos are recorded to ensure full coverage of the scene but use different camera trajectories, recording methods, and viewing angles, further increasing the difficulty of locating and matching objects across the recordings.
- Memory Update tasks assess more than coarse-grained existence and quantity.
Most previous spatial reasoning benchmarks include both coarse-grained assessments of existence and quantity and fine-grained assessments of spatial relations to evaluate model capabilities at different levels. Our Memory Update subset focuses on spatiotemporal changes between two videos, also includes these two levels of assessment:
-
Options concerning existence and quantity, such as those involving the appearance of schoolbag or the number of pillows, are relatively simple and assess the model’s ability to carefully retrieve information, localize objects, and match them across two long video sequences.
-
Options concerning changes in position and orientation, such as those involving the spatial position of the footrest or the spatial arrangement of objects on the coffee table, are more difficult and require more detailed spatial modeling and reasoning.
Both are important aspects of spatiotemporal understanding. The former focuses on coarse-grained reasoning about spatiotemporal changes, while the latter focuses on fine-grained reasoning about changes in detailed spatial states. Both require models to develop a comprehensive understanding of the spatiotemporal information in the two videos and connect this information across them, rather than rely on a few representative frames. These two types are balanced in our evaluation set to ensure that capabilities at different levels are assessed.
As reflected in our evaluation results, current models (including state-of-the-art models) still perform poorly even on questions about existence and quantity that require information retrieval, localization, and matching across two long videos. These questions remain highly challenging for current models.
Thanks again for your time and feedback. We hope these clarifications address your concerns and questions, and we welcome further feedback.
Author Response to Reviewer EzaT (1/2)
Response to Weakness1 & Question 1 (Part1)
We thank the reviewer for raising this concern. We apologize for shortening the example to fit the limited space in Figure 2, which caused the reviewer confusion about our question formulation. Below, we (1) provide a detailed clarification of the example in Figure 2, (2) explain how we ensure that questions were valid and required spatial and temporal evidence, and (3) present experimental evidence supporting the need to integrate information from both videos in Cross-Video Reasoning tasks.
1. Detailed Clarification of the example in Figure 2
The Memory Update example shown in the figure corresponds to a lengthy multiple-choice question containing a substantial amount of text. Due to the limited space available, we displayed only a simplified version of only the correct option. We apologize for the resulting ambiguity. The complete question is as follows:
<video1> <video2> These two videos show the same room recorded at two different times. Please answer the following question based on changes in the room’s spatial layout: What do you think most likely happened in this room during the interval between the two recordings?
A. Someone went to school between the two recordings, as the schoolbag was no longer present.
B. Someone used the footrest to relax between the two recordings, as spatial position of the footrest changed noticeably.
C. Someone bought new pillows for the home between the two recordings, as the number of pillows in the room increased.
D. Someone cleaned the coffee table between the two recordings, as the spatial arrangement of objects on the coffee table changed noticeably.
All questions in our benchmark are multiple-choice questions with a unique correct answer supported by concrete visual evidence and clearly distinguishable alternatives. In particular, each option in our Memory Update questions explicitly refers to specific objects in the two videos, focusing on changes in their spatial positions or poses. Answering these questions requires models to understand and connect the spatial layouts across both videos.
In this particular question, each option is a statement that refers to specific objects in the videos (the schoolbag, footrest, pillows, or objects on the coffee table) and is phrased in everyday language as much as possible. Each statement can be clearly and unambiguously judged as correct or incorrect based on the video content: the schoolbag appears in the first video but is absent from the second, making A correct, whereas the footrest’s position, the number of pillows, and the spatial arrangement of objects on the coffee table remain unchanged, making B, C, and D incorrect. Therefore, the question has a unique correct answer among the provided options, supported by explicit visual evidence.
2. How Verifying Question Validity and Non-Triviality
During dataset construction, we applied the following criteria to ensure that our questions were valid and required spatial and temporal evidence:
Eleven researchers cross-reviewed one another’s questions and annotations during dataset construction. Questions that did not satisfy these criteria were revised and reviewed again until they passed. This iterative review process was designed to ensure that our questions were valid, answerable and required non-trivial reasoning grounded in spatial and temporal evidence.
To facilitate direct inspection of the questions quality and difficulty, we provide the following anonymized links: