- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces MOMENT-VIDEO, a benchmark for evaluating whether video MLLMs can capture brief, answer-critical visual events. It has 1,000 human-verified QA pairs across 7 domains and 25 subcategories, covering four tasks: temporal ordering, temporal counting, anomaly detection and temporal reasoning. Annotators first localize the key moment and then write the question. An audit of 300 samples shows a median event duration of 0.54 s. Multiple-choice questions use shuffle-robust scoring, and open-ended answers are scored by a validated LLM judge. Across 34 models, the best model reaches 39.6%. Further analyses examine sampling rate, video duration and event coverage.
Critical strengths
- Well-motivated and under-explored target. The benchmark focuses on sub-second, answer-critical events. Its construction localizes the evidence first and then writes the question, and the event audit (median 0.54 s, 87.7% of events ≤1 s) supports the "momentary" claim.
- Broad coverage and up-to-date evaluation. It spans 7 domains, including GUI, AIGC and games, and evaluates 34 recent proprietary and open-source models.
Critical weaknesses
- Lack of blind baseline and comparable human baseline. The paper states that samples answerable without the key moment are rejected, but no quantitative result supports this. Could the authors report a text-only (blind) baseline, where models answer from the question and options alone? This would show how much of the benchmark truly depends on the momentary visual evidence. In addition, the main results lack a human baseline that can be directly compared with the models. Could the authors provide per-task human accuracy in Table 2? This would make the gap between humans and models directly visible.
- Inconsistent input settings across models. Seed-2.0 models use their native APIs with the default setting, where the model decides which and how many frames to sample. Most other models use 1 FPS for test. And sampling rate strongly affects results. Seed-2.0's leading position may therefore partly reflect its input setting. Could the authors report how many frames Seed-2.0 actually uses, and provide its results under the same 1 FPS setting for a fair comparison?
- Imbalanced task distribution. Temporal counting (TC) accounts for 472 of the 1,000 questions (47.2%), while temporal reasoning (TR) has only 85 (8.5%). Since TC is also the hardest task, the overall score is largely determined by counting performance. The paper frames the four tasks as "progressively richer capabilities", with reasoning over event-induced state changes at the top. Yet TR, the only task at this level, is the smallest subset. This imbalance may limit how accurately the benchmark measures temporal fidelity.
Questions for authors
See weaknesses above.
4 comments
Response to Reviewer rCWD part-2
W2:Seed-2.0 models use the native API with default settings, allowing the models to determine which frames to sample and how many, whereas most other evaluated models use 1 frame per second (1 FPS). Since the sampling rate strongly affects performance, Seed-2.0's advantage may partly arise from its input configuration. Could the authors report the actual number of frames used by Seed-2.0 and add results under a common 1 FPS setting for a fair comparison?
A2:Thank you for this valuable comment and for highlighting the need to evaluate Seed-2.0 under a common sampling configuration. The original experiments used the provider's default native video interface for the Seed series. Seed models are closed-source, and the existing evaluation logs do not provide the actual number of frames processed per video or the internal frame-selection rules. Therefore, the original results reflect the combined effects of each model and its provider's video preprocessing pipeline.
To address the request for a common 1 FPS setting, we reevaluated Seed-2.0-Pro, Lite, and Mini on all 1,000 questions. The client sampled at a target rate of 1 FPS with a maximum of 64 frames per video, matching the target sampling rate and frame cap used by most frame-sequence input models in the main experiments. When the number of candidate frames exceeded the budget, 64 frames were selected uniformly across the full video duration. The results are shown below:
| Model | Default native-video accuracy (%) | Accuracy at 1 FPS, max. 64 frames (%) | Change (percentage points) |
|---|---|---|---|
| Seed-2.0-Pro | 39.60 | 24.00 | -15.60 |
| Seed-2.0-Lite | 31.30 | 33.10 | +1.80 |
| Seed-2.0-Mini | 27.80 | 26.30 | -1.50 |
The effects of input configuration differ across the three Seed models. Pro shows the largest decline, Lite improves slightly, and Mini changes relatively little. Under the common 1 FPS setting, Lite ranks first, followed by Mini and Pro, whereas Pro leads under the original default configuration. This shows that the Seed series is highly sensitive to the video input configuration. Therefore, Pro's leading performance in Table 2 should be interpreted in the context of its default native video processing pipeline. These results have been added to Appendix G.4 of the revised manuscript.
Response to Reviewer rCWD part-3
W3:Of the 1000 questions, Temporal Counting (TC) accounts for 472 (47.2%), whereas Temporal Reasoning (TR) contains only 85 (8.5%). Temporal Counting is also the most difficult task, so the overall metric is largely determined by counting performance. The paper describes the four tasks as assessing progressively higher capability levels, with the highest level involving reasoning about event-induced state changes. However, the only task at this level, Temporal Reasoning (TR), has the fewest samples. This imbalance may limit the benchmark's ability to measure temporal fidelity accurately.
A3:Thank you for pointing out that the imbalanced task distribution affects the overall score. Moment-Video evaluates video multimodal large language models at different levels, from perception to understanding and reasoning, including noticing brief events, understanding how they unfold, and inferring their consequences. Temporal Counting (TC) assesses a foundational perceptual capability: recognizing and tracking each occurrence of a brief event. Missing even one occurrence can change the answer, so counting itself remains challenging. Temporal Reasoning (TR) builds on this perceptual capability and requires models to infer consequences by combining events with pre- and post-event states. As video multimodal large language models improve, we plan to expand Moment-Video with more temporal reasoning questions that require integrating brief visual events and inferring their consequences. TC contains 472 questions (47.2%), while TR contains 85 (8.5%), so TC has the largest influence on the original overall accuracy aggregated over questions. To address this concern, we additionally report macro-average accuracy, calculated as (TO + TC + AD + TR) / 4. Each task receives a weight of 25%, regardless of its sample count. The recomputed results are shown below:
| Macro rank | Model | TO (%) | TC (%) | AD (%) | TR (%) | Overall (%) | Macro (%) |
|---|---|---|---|---|---|---|---|
| 1 | Seed-2.0-Pro | 50.37 | 31.14 | 47.08 | 42.35 | 39.60 | 42.74 |
| 2 | Seed-2.0-Lite | 34.81 | 25.64 | 36.04 | 40.00 | 31.30 | 34.12 |
| 3 | Gemini-3-Flash | 41.48 | 18.43 | 31.82 | 32.94 | 26.90 | 31.17 |
| 4 | Gemini-3.1-Pro | 32.59 | 19.70 | 33.44 | 34.12 | 26.90 | 29.96 |
| 5 | Seed-2.0-Mini | 32.59 | 22.88 | 33.77 | 25.88 | 27.80 | 28.78 |
| 6 | Gemini-3.1-Flash-Lite | 34.07 | 18.64 | 30.52 | 27.06 | 25.10 | 27.57 |
| 7 | Kimi-2.6 | 25.93 | 15.89 | 36.69 | 30.59 | 24.90 | 27.28 |
| 8 | DeepSeek-v4.1-Flash | 30.37 | 14.62 | 28.90 | 31.76 | 22.60 | 26.41 |
| 9 | Qwen3.5-27B | 20.74 | 19.07 | 32.47 | 27.06 | 24.10 | 24.84 |
| 10 | Qwen3.5-397B-A17B | 27.41 | 14.83 | 32.79 | 23.53 | 22.80 | 24.64 |
| 11 | GPT-5.4 | 21.48 | 12.08 | 30.19 | 29.41 | 20.40 | 23.29 |
| 12 | Qwen3.6-27B | 19.26 | 17.16 | 31.82 | 23.53 | 22.50 | 22.94 |
| 13 | Gemma-4-31B | 30.37 | 13.56 | 24.35 | 22.35 | 19.90 | 22.66 |
| 14 | Qwen3.5-122B-A10B | 25.19 | 14.19 | 27.60 | 23.53 | 20.60 | 22.63 |
| 15 | MIMO-v2.5 | 18.52 | 17.16 | 30.84 | 23.53 | 22.10 | 22.51 |
| 16 | Qwen3.6-35B-A3B | 17.04 | 14.41 | 29.22 | 24.71 | 20.20 | 21.35 |
| 17 | Qwen3.5-35B-A3B | 20.00 | 16.10 | 26.30 | 18.82 | 20.00 | 20.31 |
| 18 | Qwen3.5-9B | 18.52 | 13.35 | 26.62 | 18.82 | 18.60 | 19.33 |
| 19 | Qwen3-VL-235B-A22B | 15.56 | 12.08 | 23.38 | 23.53 | 17.00 | 18.64 |
| 20 | Qwen3.5-4B | 19.26 | 13.77 | 21.43 | 17.65 | 17.20 | 18.03 |
| 20 | Qwen3-VL-30B-A3B | 18.52 | 13.14 | 20.45 | 20.00 | 16.70 | 18.03 |
| 22 | InternVL3.5-241B-A28B | 19.26 | 11.02 | 20.45 | 18.82 | 15.70 | 17.39 |
| 23 | InternVL3.5-8B | 15.56 | 15.04 | 20.13 | 15.29 | 16.70 | 16.51 |
| 24 | GLM-4.6V | 15.56 | 8.69 | 19.81 | 21.18 | 14.10 | 16.31 |
| 25 | LLaVA-Video-72B | 13.33 | 15.04 | 18.83 | 17.65 | 16.20 | 16.21 |
| 26 | Qwen3-VL-4B | 14.81 | 8.69 | 19.48 | 21.18 | 13.90 | 16.04 |
| 27 | Keye-VL-1.5-8B | 14.81 | 12.92 | 20.45 | 14.12 | 15.60 | 15.58 |
| 28 | InternVL3.5-30B-A3B | 12.59 | 16.10 | 21.75 | 11.76 | 17.00 | 15.55 |
| 29 | Qwen3-VL-8B | 15.56 | 10.59 | 17.86 | 16.47 | 14.00 | 15.12 |
| 30 | InternVL3.5-4B | 14.81 | 13.14 | 17.86 | 7.06 | 14.30 | 13.22 |
| 31 | GLM-4.6V-Flash | 11.11 | 7.20 | 14.29 | 20.00 | 11.00 | 13.15 |
| 32 | VITA-1.5 | 19.26 | 8.49 | 13.42 | 10.59 | 10.60 | 12.94 |
| 33 | LLaVA-Video-7B | 11.11 | 13.14 | 13.96 | 7.06 | 12.60 | 11.32 |
| 34 | VideoLLaMA3-7B | 7.52 | 13.69 | 11.72 | 8.33 | 11.80 | 10.32 |
With equal weighting of the four tasks, Seed-2.0-Pro remains first with a macro-average accuracy of 42.74%. DeepSeek-v4.1-Flash has lower original overall accuracy than Qwen3.5-27B (22.60% versus 24.10%), but higher macro-average accuracy (26.41% versus 24.84%). These results have been added to Appendix G.7 of the revised manuscript.
Dear Reviewer rCWD,
Thank you very much for your time and effort in reviewing the submissions. As the authors’ responses have been available for several days, we would greatly appreciate it if you could assess whether your concerns have been adequately addressed and update your reviews and scores.
Response to Reviewer rCWD part-1
We sincerely thank you for the time and effort you devoted to reviewing our paper and for your constructive comments. In response, we have added text-only blind baseline experiments, expanded the human baseline, added controlled sampling-rate analyses of closed-source models, and improved the balance of the evaluation metric across tasks. We hope these revisions address your concerns. The changes have been incorporated into the revised manuscript and are detailed below.
W1:(1) a text-only baseline to assess reliance on momentary visual evidence, and (2) a per-task human baseline in Table 2 to show the human–model performance gap.
A1:Thank you for these valuable comments. You raise two requests: adding a text-only baseline and including a human baseline in Table 2.
Regarding the first request: We evaluated Seed-2.0-Lite, Seed-2.0-Mini, Qwen3.6-27B, Gemma-4-31B-IT, MiMo-V2.5, and InternVL3.5-8B on all 1,000 questions in Moment-Video, providing only the question text and, for multiple-choice questions, the answer options. The table below compares the text-only results with the video-input results in Table 2. The video results retain the deployment settings used in the original paper.
Multiple-choice questions follow the benchmark's scoring rule: an answer is counted as correct only if the model selects the correct answer under every tested option order. All evaluated models score substantially lower with text-only input than with video input, supporting the practical importance of visual evidence for answering Moment-Video questions. This trend holds for both closed-source and open-weight models, indicating consistent dependence on visual input across these evaluated systems. The difference between multiple-choice and open-ended questions also helps explain why some questions are still answered correctly without video. Multiple-choice questions directly expose candidate answers, allowing models to use language priors or eliminate implausible options. In addition, with the model temperature set to 0, models may more readily produce consistent answers across shuffled option orders, which is one reason for the higher multiple-choice scores. Open-ended questions instead require generating specific event counts, actions, or state changes. Their lower text-only accuracy is consistent with these details being difficult to recover from the question alone; remaining correct answers may arise from linguistic cues or guessing. These results have been added to Appendix G.6 of the revised manuscript.
Regarding the second request: Appendix E (Table 8) of the original paper reported a human solvability study in which participants answered questions after watching the corresponding videos. To provide a more detailed human reference, we further aggregated the original response records by the four subtasks and seven video domains and will add the corresponding results to Table 2. Each of the 15 retained participants answered 20 questions, yielding 300 responses to 200 sampled questions. Overall accuracy was 84.33%, and individual participant accuracies ranged from 70.00% to 95.00%. The following tables also include Seed-2.0-Pro, the best-performing video model overall in Table 2, as an overall reference. All values are accuracies (%).
These breakdowns show that, in the original sampled study, humans achieved accuracy above 80% on all four subtasks and answered most questions correctly across the seven video domains. This provides further evidence that the sampled questions are solvable when the videos are available. Compared with the human reference, video models still have substantial room for improvement on the full benchmark. The human baseline is now included in Table 2, and the related details have been added to Appendix E of the revised manuscript.