acceptodds
Under review as a conference paper at ICLR 2027

MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence

Abstract

Spatial understanding over continuous visual input is crucial for MLLMs to evolve into general-purpose assistants in physical environments. Yet there is still no comprehensive benchmark that holistically assesses the progress toward this goal. In this work, we introduce MMSI-Video-Bench, a fully human-annotated benchmark for video-based spatial intelligence in MLLMs. It operationalizes a four-level framework, Perception, Planning, Prediction, and Cross-Video Reasoning, through 1,106 questions grounded in 1,278 clips from 25 datasets and in-house videos. Each item is carefully designed and reviewed by 3DV experts with explanatory rationales to ensure precise, unambiguous grounding. Leveraging its diverse data sources and holistic task coverage, MMSI-Video-Bench also supports three domain-oriented sub-benchmarks (Indoor Scene Perception Bench, Robot Bench and Grounding Bench) for targeted capability assessment. We evaluate 25 strong open-source and proprietary MLLMs, revealing a striking human–AI gap: many models perform near chance, and the best reasoning model lags humans by nearly 60%. We further find that spatially fine-tuned models still fail to generalize effectively on our benchmark. Fine-grained error analysis exposes systematic failures in geometric reasoning, motion grounding, long-horizon prediction, and cross-video correspondence. We also show that typical frame-sampling strategies transfer poorly to our reasoning-intensive benchmark, and that neither 3D spatial cues nor chain-of-thought prompting yields meaningful gains. We expect our benchmark to establish a solid testbed for advancing video-based spatial intelligence.

Primary area:
datasets and benchmarks
Keywords:
Spatial Intelligence, Video-Based Benchmark, 3D Understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EzaTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MMSI-Video-Bench, a human-annotated benchmark containing 1,106 multiple-choice questions grounded in 1,278 video clips from 25 public datasets and newly recorded videos. It covers spatial construction, motion understanding, planning, prediction, and cross-video reasoning, with questions and explanatory rationales developed and reviewed by 3D vision researchers. The authors evaluate 25 open-source and proprietary models under two frame-input settings, finding low overall accuracy and a substantial gap from the reported human baseline. An analysis of 520 incorrect responses further characterizes failures in visual grounding, geometric reasoning, instruction interpretation, and cross-video inference.

Critical strengths
  1. The benchmark combines diverse video sources with expert-written questions, explanatory rationales, and cross-review, providing a valuable alternative to narrowly scoped benchmarks built through automatic templates.
  2. Its coverage of spatial, temporal, and cross-video tasks enables a broader assessment of current models, while subtype-level results reveal weaknesses that aggregate video-understanding scores can obscure.
Critical weaknesses
  1. The paper does not establish how strongly the questions depend on the spatial and temporal evidence they are intended to test. For example, the memory-update question in Figure 2 labels “Someone went to school” as the answer to an unobserved event between recordings, but the presented example does not explain why this conclusion is uniquely supported over other plausible explanations. Expert agreement alone does not resolve the distinction between a visually grounded answer and a shared commonsense assumption. The main-text experiments also lack controls that remove temporal information or one of the videos, leaving it unclear whether the proposed categories consistently require their claimed capabilities.
  2. Section 4.3 interprets poor performance by spatially adapted models as evidence of limited spatial generalization, even though it explicitly identifies degraded instruction following as a major source of failure. VLM3R’s accuracy of 4.97%, far below the approximately 24% random baseline, makes separating these factors particularly important. Without distinguishing invalid or noncompliant responses from valid but spatially incorrect answers, the results cannot establish whether the learned spatial capabilities themselves fail to transfer. Similarly, the error analysis infers limitations in temporal aggregation and internal representations from response-level error labels without directly testing those explanations, weakening the benchmark’s claimed diagnostic value.
Questions for authors
  1. What specific visual evidence supports “Someone went to school” in Figure 2 over the alternative answer choices, and how were questions about unobserved events checked to ensure that the correct answer was supported by the videos rather than primarily by commonsense assumptions?
  2. For Spatial-MLLM and VLM3R, what proportion of failures comes from instruction following or answer formatting, and what proportion consists of valid responses with incorrect spatial reasoning? Does the conclusion about poor spatial generalization still hold when these failure types are evaluated separately?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer EzaT (1/2)

Response to Weakness1 & Question 1 (Part1)

We thank the reviewer for raising this concern. We apologize for shortening the example to fit the limited space in Figure 2, which caused the reviewer confusion about our question formulation. Below, we (1) provide a detailed clarification of the example in Figure 2, (2) explain how we ensure that questions were valid and required spatial and temporal evidence, and (3) present experimental evidence supporting the need to integrate information from both videos in Cross-Video Reasoning tasks.

1. Detailed Clarification of the example in Figure 2

The Memory Update example shown in the figure corresponds to a lengthy multiple-choice question containing a substantial amount of text. Due to the limited space available, we displayed only a simplified version of only the correct option. We apologize for the resulting ambiguity. The complete question is as follows:

<video1> <video2> These two videos show the same room recorded at two different times. Please answer the following question based on changes in the room’s spatial layout: What do you think most likely happened in this room during the interval between the two recordings?

A. Someone went to school between the two recordings, as the schoolbag was no longer present.

B. Someone used the footrest to relax between the two recordings, as spatial position of the footrest changed noticeably.

C. Someone bought new pillows for the home between the two recordings, as the number of pillows in the room increased.

D. Someone cleaned the coffee table between the two recordings, as the spatial arrangement of objects on the coffee table changed noticeably.

All questions in our benchmark are multiple-choice questions with a unique correct answer supported by concrete visual evidence and clearly distinguishable alternatives. In particular, each option in our Memory Update questions explicitly refers to specific objects in the two videos, focusing on changes in their spatial positions or poses. Answering these questions requires models to understand and connect the spatial layouts across both videos.

In this particular question, each option is a statement that refers to specific objects in the videos (the schoolbag, footrest, pillows, or objects on the coffee table) and is phrased in everyday language as much as possible. Each statement can be clearly and unambiguously judged as correct or incorrect based on the video content: the schoolbag appears in the first video but is absent from the second, making A correct, whereas the footrest’s position, the number of pillows, and the spatial arrangement of objects on the coffee table remain unchanged, making B, C, and D incorrect. Therefore, the question has a unique correct answer among the provided options, supported by explicit visual evidence.

2. How Verifying Question Validity and Non-Triviality

During dataset construction, we applied the following criteria to ensure that our questions were valid and required spatial and temporal evidence:

  • A unique, visually supported answer. Each question had to have a unique correct answer among the provided options, identifiable from the video evidence alone. The alternatives had to be clearly distinguishable and unambiguous. This requirement ensures that the correct answer is supported by video evidence rather than primarily by common-sense assumptions, directly addressing the reviewer’s concern.
  • Non-trivial spatial and temporal reasoning. To ensure that the questions depended on spatial and temporal evidence, we required that:
    • The questions could not be answered solely based on surface-level 2D semantic information but had to involve the use of 3D spatial information.
    • The questions could not be resolved by referring to a single key frame or a short temporal window; instead, they had to require information retrieval and integration across a broader temporal range.

Eleven researchers cross-reviewed one another’s questions and annotations during dataset construction. Questions that did not satisfy these criteria were revised and reviewed again until they passed. This iterative review process was designed to ensure that our questions were valid, answerable and required non-trivial reasoning grounded in spatial and temporal evidence.

To facilitate direct inspection of the questions quality and difficulty, we provide the following anonymized links:

AuthorsOpenReview ↗

Author Response to Reviewer EzaT (2/2)

Response to Weakness1 & Question 1 (Part2)

3. The need to integrate information from both videos in Cross-Video Reasoning tasks

The frame-sampling experiments in Appendix C.2 (Frame Sampling Study) demonstrate the importance of video-derived visual spatial information for model performance on MMSI-Video. During the rebuttal period, we additionally conducted additional ablation experiments to demonstrate that answering Cross-Video Reasoning questions requires integrating spatiotemporal information from both videos:

When one video was removed, all four representative models (o3, GPT-4o, Gemini-2.5-Flash, Qwen2.5-VL-7B, InternVL2.5-38B ) in the table showed substantial performance drops, with their performance approaching the level of random guessing. These results demonstrate the importance of spatiotemporal information in these tasks.

ModelBoth videos (%)One video (%)
o334.929.7 (−5.2)
GPT-4o29.625.6 (−4.0)
Gemini-2.5-Flash34.331.4 (−2.9)
Qwen2.5-VL-7B26.723.3 (−3.4)
InternVL2.5-38B32.627.9 (−4.1)

For Weakness 2 & Question 2

As mentioned in our paper, these models’ capabilities do not generalize effectively to other benchmarks, and their original abilities may even be impaired. One of the affected abilities is instruction following.

In fact, when evaluating these models, to mitigate this effect and make the evaluation as effective as possible, we used an answer-extraction method designed to be as robust as possible to extract valid answers wherever possible. As a result, all responses from SpaceQwen and nearly all responses from Spatial-MLLM were valid, so their evaluation scores can directly reflect the spatial capabilities learned and transferred by these models. In contrast, VLM3R exhibited severe degradation in instruction-following ability, with most responses consisting of repetitive numerical outputs, resulting in an abnormally low score. Its evaluation score was clearly affected by its instruction-following ability. Furthermore, to provide additional supporting evidence, we evaluated larger spatially fine-tuned models, RoboBrain 2.0 (32B) and SpatialThinker (30B). Their responses were entirely valid, so their scores can directly reflect the spatial capabilities learned and transferred by these models.

MethodAcc. (vs. baseline)Failure Rate
SpaceQwen (3B)27.58 (−0.09)0/1106
Spatial-MLLM (7B)24.05 (−4.43)5/1106
VLM3R (7B)4.97 (−23.51)905/1106
RoboBrain 2.0 (32B)30.04 (+1.40)0/1106
SpatialThinker (30B)30.11 (+1.31)0/1106

As can be seen, only VLM3R is affected by invalid or non-instruction-following responses. For the other models, their evaluation scores can directly reflect the spatial capabilities they have learned and transferred, with essentially no influence from invalid or non-instruction-following responses. Compared with their corresponding baselines, these models show little to no performance improvement on MMSI-Video-Bench, indicating that their capabilities do not generalize effectively to other benchmarks. After accounting for non-instruction-following responses, the conclusion of limited spatial generalization still holds.

Thank you for your response and clarification. My remaining concern is whether the Memory Update questions require temporal reasoning beyond comparing two snapshots. In Figure 2, a model could select A simply by noticing that the schoolbag is gone, without inferring what happened between the recordings. Removing one video also prevents that comparison, so the drop in accuracy leaves open whether one representative frame from each video would suffice. This distinction matters for interpreting what a correct answer tells us about the model’s capabilities.

We thank the reviewer for the follow-up and constuctive feeback. Below, we explain why one representative frame from each video is insufficient to answer this example and clarify the different levels of spatiotemporal understanding assessed by our Memory Update tasks.

  1. One representative frame from each is insufficient.

As stated in our rebuttal, our question-design criteria ensure non-trivial spatial and temporal reasoning: “The questions could not be resolved by referring to a single key frame or a short temporal window.” This requirement also applies to the example in Figure 2.

The schoolbag is not necessarily present in the first video (if it is absent, the option is also incorrect). Furthermore, its absence from its original location in the second video may may be because it has disappeared from the scene or has been moved to another location within the same scene (both scenarios are possible in our tasks). From a representative frame, a model can only determine that the schoolbag is no longer at its original location.It cannot determine whether the schoolbag has disappeared from the scene or remains elsewhere within it.

Answering the question therefore requires careful information retrieval, object localization, and matching across both long video sequences rather than from two representative frames. (The schoolbag might be visible in a corner of a particular frame, or it might be absent from the scene altogether) Moreover, both videos are recorded to ensure full coverage of the scene but use different camera trajectories, recording methods, and viewing angles, further increasing the difficulty of locating and matching objects across the recordings.

  1. Memory Update tasks assess more than coarse-grained existence and quantity.

Most previous spatial reasoning benchmarks include both coarse-grained assessments of existence and quantity and fine-grained assessments of spatial relations to evaluate model capabilities at different levels. Our Memory Update subset focuses on spatiotemporal changes between two videos, also includes these two levels of assessment:

  • Options concerning existence and quantity, such as those involving the appearance of schoolbag or the number of pillows, are relatively simple and assess the model’s ability to carefully retrieve information, localize objects, and match them across two long video sequences.

  • Options concerning changes in position and orientation, such as those involving the spatial position of the footrest or the spatial arrangement of objects on the coffee table, are more difficult and require more detailed spatial modeling and reasoning.

Both are important aspects of spatiotemporal understanding. The former focuses on coarse-grained reasoning about spatiotemporal changes, while the latter focuses on fine-grained reasoning about changes in detailed spatial states. Both require models to develop a comprehensive understanding of the spatiotemporal information in the two videos and connect this information across them, rather than rely on a few representative frames. These two types are balanced in our evaluation set to ensure that capabilities at different levels are assessed.

As reflected in our evaluation results, current models (including state-of-the-art models) still perform poorly even on questions about existence and quantity that require information retrieval, localization, and matching across two long videos. These questions remain highly challenging for current models.

Thanks again for your time and feedback. We hope these clarifications address your concerns and questions, and we welcome further feedback.

Reviewer_XHZsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces MMSI-Video-Bench, a human-annotated benchmark for evaluating spatial intelligence in multimodal large language models from video inputs. It contains 1,106 multiple-choice questions grounded in 1,278 video clips drawn from 25 public datasets and additional in-house recordings. The benchmark covers five categories—spatial construction, motion understanding, planning, prediction, and cross-video reasoning—organized into 13 subtypes. Questions, answer options, and explanatory rationales are manually designed and cross-reviewed by researchers specializing in 3D vision. The paper evaluates 25 open-source and proprietary models under uniform frame sampling and sufficient-coverage settings, reporting a substantial gap between model and human performance. Additional experiments examine spatially fine-tuned models, frame-sampling strategies, auxiliary 3D cues, and chain-of-thought prompting. An analysis of 520 incorrect responses further characterizes task-specific failure patterns, positioning the benchmark as a diagnostic resource for video-based spatial reasoning.

Critical strengths
  1. Broad coverage of video-based spatial reasoning. The benchmark extends evaluation beyond scene-layout understanding to motion, planning, prediction, and cross-video reasoning, with 13 subtypes grounded in diverse video sources. The inclusion of memory updates and multi-view integration is particularly valuable for assessing how models combine observations across time and viewpoints. This broader scope makes the benchmark consequential as a complementary test of capabilities that narrower spatial benchmarks may overlook.

  2. Substantial expert curation and supporting validation. Questions, distractors, and explanatory rationales are manually designed and cross-reviewed by 11 researchers specializing in 3D vision. This process supports flexible question formulations and careful verification of answer grounding, reducing reliance on fixed generation templates. The repeated-evaluation and bootstrap analyses provide additional evidence for overall score and ranking stability. Together, these efforts strengthen the benchmark’s potential value as a reliable evaluation resource.

Critical weaknesses
  1. Insufficient validation of the specific capabilities measured by the benchmark. The frame-count and sampling experiments provide useful evidence that broader visual coverage improves performance. However, they do not fully distinguish temporal reasoning and cross-video integration from simply accessing more static visual information. Targeted controls—such as text-only evaluation, temporal perturbations with timestamps appropriately controlled, and single-video inputs for cross-video questions—would help establish which capabilities are necessary for answering each task category. This matters because the benchmark’s diagnostic value depends on whether its categories reliably measure the claimed abilities.

  2. Incomplete characterization of frontier-model performance under adequate inference budgets. Although the evaluation includes strong models, the best-performing model, Gemini 3 Pro, is reported only under Uniform-50. It therefore remains unclear how much of the reported human–model gap reflects spatial reasoning limitations versus restricted visual coverage or inference settings. Reporting exact model versions, evaluation dates, and inference budgets, together with larger input budgets where supported and stronger publicly available models where applicable, would better substantiate the conclusions about current model capabilities and the benchmark’s remaining difficulty.

Questions for authors
  1. Could the authors provide targeted controls on representative subsets to validate the capabilities required by each task category? In particular, text-only evaluation, temporal perturbations with timestamps appropriately controlled, and single-video inputs for cross-video questions would help determine whether performance depends on the claimed temporal and cross-video reasoning abilities.

  2. Could the authors clarify the exact model versions, evaluation dates, and inference budgets, and supplement the strongest models’ results with larger frame coverage or reasoning budgets where supported? Evaluating additional frontier models publicly available at the evaluation date would also help determine whether the reported human–model gap persists under stronger model and inference settings.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer XHZs (1/2)

For Weakness 1 & Question 1

We thank the reviewer for suggesting these targeted control experiments. We also believe that these experiments are necessary to assess whether model performance depends on the temporal and cross-video reasoning abilities being evaluated. We selected five representative models (o3, GPT-4o, Gemini-2.5-Flash, Qwen2.5-VL-7B, and InternVL2.5-38B) and conducted the following experiments: (a) text-only evaluation, (b) perturbation of the temporal order of video frames, and (c) providing only one video for cross-video questions. We compared the models’ performance under these settings.

ModelOverall BaselineText-onlyTemporal ShufflingCross-Video BaselineCross-Video: One Video
o337.023.4 (−13.6)30.2 (−6.8)34.929.7 (−5.2)
GPT-4o31.621.0 (−10.6)27.1 (−4.5)29.625.6 (−4.0)
Gemini-2.5-Flash35.427.8 (−7.6)28.7 (−6.7)34.331.4 (−2.9)
Qwen2.5-VL-7B29.416.7 (−12.7)26.4 (−3.0)26.723.3 (−3.4)
InternVL2.5-38B31.027.5 (−3.5)28.8 (−2.2)32.627.9 (−4.7)

As shown, under the text-only and temporal-perturbation settings, all models exhibited clear performance declines, with performance even approaching the level of random guessing. This demonstrates the importance of visual cues and temporal information for model performance on this benchmark. For Cross-Video Reasoning questions, removing one video also caused clear performance declines across all models, demonstrating that reasoning over the spatiotemporal information from both videos is necessary for these tasks.

AuthorsOpenReview ↗

Author Response to Reviewer XHZs (2/2)

For Weakness 2 & Question 2

We thank the reviewer for highlighting the need for more detailed evaluation reporting and further evidence on whether the human–model gap persists with stronger models and more comprehensive evaluation settings. Below, we provide additional evaluation details, including model versions and reasoning settings, and report further results on the human–model gap with stronger models and more comprehensive evaluation settings.

1. Detailed model evaluation information.

  • Exact model versions. The table below provides the exact model versions used in our evaluation. For closed-source models, we provide the specific model IDs; for open-source models, we provide the Hugging Face repository names containing the model weights. All model evaluations were conducted after December 2025.
Name in the PaperModel Version
O4-mini / O3 / GPT-4o / GPT-5o4-mini-2025-04-16 / o3-2025-04-16 / gpt-4o-2024-11-20 / gpt-5-2025-08-07
Claude-haiku-4.5claude-haiku-4-5-20251001
Gemini 3 Pro / Gemini 2.5 Flashgemini-3-pro-preview / gemini-2.5-flash
Doubao-1.5-thinking / Seed-1.6-visiondoubao-1-5-thinking-vision-pro-250428 / doubao-seed-1-6-vision-250815
InternVL2.5-8B/38B/78BOpenGVLab/InternVL2_5-{}B
InternVL3-8B/38B/78BOpenGVLab/InternVL3-{}B
InternVideo2.5-8BOpenGVLab/InternVideo2_5_Chat_8B
LLaVA-Video-7B/72Blmms-lab/LLaVA-Video-{}B-Qwen2
QwenVL2.5-7B/32B/72BQwen/Qwen2.5-VL-{}B-Instruct
QwenVL3-8B/30BQwen/Qwen3-VL-8B-Instruct / Qwen/Qwen3-VL-30B-A3B-Instruct

{} denotes the corresponding model size listed in the left column.

  • Reasoning settings. To evaluate the models’ strongest capabilities, we used the highest available reasoning setting for all closed-source models with a thinking mode. Specifically, we set reasoning_effort="high" for GPT-5, o3, and o4-mini; thinking_level="high" for Gemini 3 Pro; and thinking_budget=24576 for Gemini 2.5 Flash.

2. Human–model gap persists under stronger & frontier model and more comprehensive evaluation settings.

We thank the reviewer for the suggestion. Due to access limitations, we were not able to evaluate the latest models at the submission stage. We now additionally report results on GPT-5.5, Gemini 3.5-Flash, and Qwen3.6, which represent the most advanced models currently accessible to us. We will also continue updating our benchmark leaderboard.

For these three models, we evaluated performance under both the Uniform-50 and Sufficient-Coverage settings, with Sufficient-Coverage providing the most comprehensive visual information. The reasoning mode was set to high for all three models to ensure that the results reflected their strongest performance. The results are as follows:

ModelSettingOverallSpatial ConstructionMotion UnderstandingPlanningPredictionCross-Video
GPT-5.5Uniform-5052.2660.1348.4847.5839.0246.51
GPT-5.5Sufficient-Coverage52.3556.0353.4151.6143.9045.35
Gemini 3.5-FlashUniform-5049.4655.1746.2147.5836.5946.51
Gemini 3.5-FlashSufficient-Coverage51.6356.6848.8649.1941.4648.84
Qwen3.6-35B-A3BUniform-5035.0336.2542.9823.4038.3631.88
Qwen3.6-35B-A3BSufficient-Coverage34.1138.3236.3626.6030.2630.19

We are pleased to observe that, compared with previous models from the same families, these recent models show particularly significant improvements in spatial intelligence, thereby narrowing the gap to human performance. Currently the strongest model GPT-5.5, using the highest reasoning setting and the most comprehensive visual input, achieved the highest benchmark score of 52.35. Nevertheless, there remains a substantial gap of 45% compared with human performance, indicating that there is still considerable room for improvement. At the same time, these new results do not affect the core conclusions of our work. In the final version, we will include these results and revise the related discussions accordingly.

I thank the authors for their response and the additional experiments, which address most of my questions. After considering the rebuttal, I will maintain my original rating of 3 (Weak Acceptance).

Thank you very much for your time and thoughtful feedback. We are glad that our response have addressed most of your questions, and we appreciate your positive assessment of our work.

Reviewer_vqivOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MMSI-Video-Bench contains 1,106 human-annotated multiple-choice questions across five categories and 13 subtypes, using 1,278 clips from 25 public datasets and 140 in-house recordings. The authors test 25+ MLLMs under Uniform-50 and Sufficient-Coverage. Gemini 3 Pro performs best at 38.0%, against a reported human score of 96.4% and chance accuracy of 24.1%. The analyses identify geometric reasoning as a common bottleneck and report little benefit from spatial fine-tuning, AKS sampling, VGGT-derived cues, or CoT prompting.

Critical strengths
  1. Coverage of Planning, Prediction, and Cross-Video Reasoning extends beyond common spatial recognition tasks, while Table 5 supports greater question diversity than existing benchmarks.

  2. The benchmark provides a useful challenge for current models. The best model reaches 38.0%, and the frame-sampling experiments suggest that answering many questions requires information spread across the video. This makes the dataset useful for testing video spatial reasoning.

Critical weaknesses
  1. The human–model comparison is not sufficiently documented. The paper does not specify whether humans saw the same frames as models or had access to full videos, replay, or scrubbing. Uniform-50 may omit evidence needed to answer some questions, so the reported gap could reflect both reasoning ability and unequal information access.

  2. The diagnostic conclusions need stronger support. Several findings repeat prior work, while the error analysis lacks details about annotators and agreement. The evaluation of spatially trained models is also limited to 3B/7B models, which provides limited support for broad conclusions about generalization after spatial fine-tuning.

Questions for authors
  1. Did humans and models have access to the same visual information?
  2. How accurate are the VGGT-generated 3D cues? When the cues are accurate, do they improve model performance?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer vqiv (1/2)

Response to Weakness 1 & Question 1

We thank the reviewer for highlighting the need to clarify the visual information available to human participants and models. We agree that this is essential for interpreting the reported human–model performance gap. Below, we specify the human viewing conditions and explain why our evaluation protocol provides a fair basis for comparing human and model performance. The human-evaluation protocol was as follows:

  1. Human evaluation followed the same Sufficient-Coverage Setting as model evaluation, as defined in Section 4.1. Specifically, participants were not given the full original videos. Instead, they received exactly the same frame sequences used as model inputs under this Sufficient-Coverage setting, ensuring coverage of all key visual information required to answer each question. We also provided a convenient frame-browsing interface that allowed participants to quickly find and locate frames at specific time points. This design was intended to ensure fairness: human participants and models received exactly the same visual input.
  2. Participants were allowed to answer without a time limit and could repeatedly review the key frames during reasoning. We consider this a fair setting, as models can also attend to and re-examine the provided key frames during inference, particularly strong reasoning models such as o3 and Gemini.

Most models evaluated in our paper support the Sufficient-Coverage setting so they were evaluated using exactly the same frame sequences shown to the human participants. The corresponding human–model comparisons are therefore based on equal access to visual information. While remaining models cannot support this setting due to their input limitations, we therefore introduced Uniform-50 as an additional evaluation setting to accommodate these models and include them in the comparison. In the benchmark release, we will recommend prioritizing the Sufficient-Coverage Setting to ensure that models have access to all visual information needed to answer each question like human evalutors.

We will incorporate these details into the final version to clarify the human-evaluation protocol and better support the analysis of the human–AI performance gap.

Response to Weakness 2 (Part 1)

We thank the reviewer for raising concerns about the novelty of our insights, the methodological details of our error analysis, and the scope of the conclusions drawn from our evaluation of models trained on spatial tasks. We address each concern in turn below.

  1. (For Weakness 2-a) the Novelty of MMSI-Video-Bench and Diagnostic Insights

Given the shared focus on spatial intelligence, some of our findings are consistent with those of previous spatial-intelligence benchmarks. More importantly, while existing benchmarks are largely limited to specific task types, we provide a unified and holistic evaluation framework covering different dimensions of spatial intelligence, including task types that have rarely been explored in previous benchmarks, thereby enabling new insights not covered by prior work. Our definition and taxonomy of video-level spatial intelligence are holistic and hierarchical, comprising the following levels:

  • (i) Understanding and reasoning over intrinsic spatiotemporal information within a video.
  • (ii) High-level reasoning based on video spatiotemporal information.
  • (iii) Spatiotemporal reasoning across multiple video clips.

Existing benchmarks are largely confined to specific task types (as shown in Table 1). The community still lacks a comprehensive benchmark covering the full range of tasks in spatial intelligence. MMSI-Video-Bench fills this gap by providing complete task coverage and a diagnostic evaluation tool for analyzing model capabilities in real-world spatial intelligence.

A. This comprehensive taxonomy encompasses many capabilities that previous spatial-reasoning benchmarks have not systematically evaluated. For example, the Interactive Motion subtask within Motion Understanding evaluates interactive motion between different instances; Spatio-Temporal Prediction requires models to reason about and predict future spatial states based on existing spatiotemporal information; and Memory Update requires models to associate multiple video clips from different time periods and continuously update their understanding of the scene accordingly.

B. Through this more comprehensive task design and the corresponding error analysis, we derived several novel insights. As reported in Error Statistics, our findings on Spatial Construction are consistent with those of previous benchmarks. More importantly, for task families that have rarely been explored in prior work, such as Prediction & Planning Tasks and Cross-Video Reasoning Tasks, we identify key capability bottlenecks in current models and provide targeted suggestions for future improvement.

AuthorsOpenReview ↗

Author Response to Reviewer vqiv (2/2)

Response to Weakness 2 (Part 2)

  1. (For Weakness 2-b) the Annotation Protocol for Error Analysis

We provide further details on the annotation procedure underlying our error analysis. Before the main annotation stage, we conducted a pilot annotation of representative failure cases from selected models. This stage served two purposes: (i) to establish the error categories and their definitions for subsequent annotation, and (ii) to check consistency in how annotators applied these definitions. We found that, when following the strict definitions of each error category, the annotators’ classifications were consistent across all but a very small number of pilot cases. For these few cases, consensus was also reached through discussion.

During the main annotation stage, five experts with backgrounds in 3D vision examined the models’ reasoning outputs and classified the errors using the established categories. Each expert was assigned a separate subset of questions. A sixth expert then conducted a final review to check the accuracy of the annotations. Any disagreements arising during this review were resolved through discussion among the annotators and the reviewing expert.

  1. (For Weakness 2-c) Evaluating the Generalization of Larger Spatially Fine-Tuned Models

We have extended our evaluation to include two larger spatially fine-tuned models, RoboBrain 2.0 (32B) and SpatialThinker (30B). As shown below, across the evaluated model scales, improvements over the corresponding baselines remain modest(<1.4) while several models exhibit performance declines.

ModelAcc. (%)Δ (pp)
SpaceQwen (3B)27.58−0.09
Spatial-MLLM (7B)24.05−4.43
VLM3R (7B)4.97−23.51
RoboBrain 2.0 (32B)30.04+1.40
SpatialThinker (30B)30.11+1.31

Δ denotes the change relative to the corresponding baseline, in percentage points.

These results provide further evidence that the spatial fine-tuning used by these models does not consistently translate into improved performance on MMSI-Video, even at larger model scales, highlighting the challenge of transferring spatial capabilities to the tasks covered by our benchmark.

Response to Question 2

Our goal was to provide models with explicit 3D spatial cues to support spatial reasoning. We selected VGGT as a representative and capable 3D reconstruction method available at the time, capable of deriving such cues from the existing visual input without requiring additional 3D information.

  1. Reliability and task-specific usability of the generated cues. As discussed in Appendix D.1, VGGT sometimes produces fragmented geometry in the complex multi-room, multi-floor, and dynamic scenes included in MMSI-Video-Bench. We therefore manually reviewed the generated cues using three criteria: whether they were visually plausible, useful for answering the corresponding questions, and able to make explicit spatial information that was not readily apparent in the original 2D videos. Actually 39.8% met these criteria.

  2. Performance gains remain limited even on the subset with usable cues. On the subset that passed our manual review, the average score improvement over evaluation without 3D cues was 2.3% less than 3%. Models often partially ignored the cues or incorrectly associated them with the video content and the question, despite explicit instructions encouraging their use. Our case-level analysis suggests that using these cues still requires basic spatial reasoning abilities. A single cue image rarely contains all the information needed to answer a question. Although the 3D cues simplify some aspects of reasoning, models still need to integrate information and reason across multiple images (original ones and cues). In most cases, the models failed to follow a correct reasoning path. These observations suggest that limitations in the models’ own spatial reasoning abilities prevent them from fully exploiting the information provided by the 3D cues.

As discussed in Appendix D.1, without dedicated training on spatial data or the incorporation of 3D architectures, owing to limitations in relative reasoning abilities, current models struggle to achieve substantial performance gains from 3D cues that seem useful to humans.

Thank you for your response. I think it addressed most of my questions. I also think this is a valuable dataset, especially after seeing the results for stronger models in the rebuttal. I will consider raising my score.

Thank you very much for your time and constructive feedback. We are glad that our responses have addressed most of your concerns, and we appreciate your recognition of the dataset’s value. We would be grateful if these clarifications and additional results could be reflected in the final assessment. Please feel free to let us know if you have any further questions and we would be happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.