acceptodds
Under review as a conference paper at ICLR 2027

Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events

Abstract

Video multimodal large language models (MLLMs) have made rapid progress in general and long-form video understanding, yet their ability to capture and use brief, answer-critical visual evidence remains underexplored. Momentary actions or state transitions may determine an answer while occupying only a small fraction of the video. Performance on broad video understanding tasks alone does not establish whether models can reliably identify and use this temporally localized evidence. We introduce Moment-Video, a benchmark for evaluating temporal fidelity through momentary visual event understanding. Questions are constructed around visually observable events or event sequences, requiring models to notice, count, describe, or reason about transient evidence. The benchmark contains 1,000 human-verified video-QA pairs across seven domains and 25 subcategories, covering four tasks: Temporal Occurrence, Temporal Counting, Action Description, and Temporal Reasoning. We evaluate 34 proprietary and open-source MLLMs. Under the main evaluation settings, the best-performing model, Seed-2.0-Pro, achieves 39.6% overall accuracy, while most open-source models remain below 25%. Diagnostic analyses show that denser frame sampling improves some models, but substantial errors persist even when sampled frames cover all annotated necessary event intervals in an audited subset. Performance also declines on longer videos, suggesting additional temporal-localization challenges. By exposing these limitations, Moment-Video provides a diagnostic benchmark to guide the evaluation and development of video MLLMs that more reliably understand brief, answer-critical visual events.

Primary area:
datasets and benchmarks
Keywords:
Video Multimodal Large Language Models, Video Understanding Multimodal Benchmark, Temporal Perception, Momentary Visual Events, Frame Sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rCWDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MOMENT-VIDEO, a benchmark for evaluating whether video MLLMs can capture brief, answer-critical visual events. It has 1,000 human-verified QA pairs across 7 domains and 25 subcategories, covering four tasks: temporal ordering, temporal counting, anomaly detection and temporal reasoning. Annotators first localize the key moment and then write the question. An audit of 300 samples shows a median event duration of 0.54 s. Multiple-choice questions use shuffle-robust scoring, and open-ended answers are scored by a validated LLM judge. Across 34 models, the best model reaches 39.6%. Further analyses examine sampling rate, video duration and event coverage.

Critical strengths
  1. Well-motivated and under-explored target. The benchmark focuses on sub-second, answer-critical events. Its construction localizes the evidence first and then writes the question, and the event audit (median 0.54 s, 87.7% of events ≤1 s) supports the "momentary" claim.
  2. Broad coverage and up-to-date evaluation. It spans 7 domains, including GUI, AIGC and games, and evaluates 34 recent proprietary and open-source models.
Critical weaknesses
  1. Lack of blind baseline and comparable human baseline. The paper states that samples answerable without the key moment are rejected, but no quantitative result supports this. Could the authors report a text-only (blind) baseline, where models answer from the question and options alone? This would show how much of the benchmark truly depends on the momentary visual evidence. In addition, the main results lack a human baseline that can be directly compared with the models. Could the authors provide per-task human accuracy in Table 2? This would make the gap between humans and models directly visible.
  2. Inconsistent input settings across models. Seed-2.0 models use their native APIs with the default setting, where the model decides which and how many frames to sample. Most other models use 1 FPS for test. And sampling rate strongly affects results. Seed-2.0's leading position may therefore partly reflect its input setting. Could the authors report how many frames Seed-2.0 actually uses, and provide its results under the same 1 FPS setting for a fair comparison?
  3. Imbalanced task distribution. Temporal counting (TC) accounts for 472 of the 1,000 questions (47.2%), while temporal reasoning (TR) has only 85 (8.5%). Since TC is also the hardest task, the overall score is largely determined by counting performance. The paper frames the four tasks as "progressively richer capabilities", with reasoning over event-induced state changes at the top. Yet TR, the only task at this level, is the smallest subset. This imbalance may limit how accurately the benchmark measures temporal fidelity.
Questions for authors

See weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer rCWD part-1

We sincerely thank you for the time and effort you devoted to reviewing our paper and for your constructive comments. In response, we have added text-only blind baseline experiments, expanded the human baseline, added controlled sampling-rate analyses of closed-source models, and improved the balance of the evaluation metric across tasks. We hope these revisions address your concerns. The changes have been incorporated into the revised manuscript and are detailed below.

W1:(1) a text-only baseline to assess reliance on momentary visual evidence, and (2) a per-task human baseline in Table 2 to show the human–model performance gap.

A1:Thank you for these valuable comments. You raise two requests: adding a text-only baseline and including a human baseline in Table 2.

Regarding the first request: We evaluated Seed-2.0-Lite, Seed-2.0-Mini, Qwen3.6-27B, Gemma-4-31B-IT, MiMo-V2.5, and InternVL3.5-8B on all 1,000 questions in Moment-Video, providing only the question text and, for multiple-choice questions, the answer options. The table below compares the text-only results with the video-input results in Table 2. The video results retain the deployment settings used in the original paper.

ModelVideo-input accuracy (%)Text-only accuracy (%)Drop (percentage points)Text-only multiple-choice (%)Text-only open-ended (%)
Seed-2.0-Lite31.3014.6016.7026.5810.88
Seed-2.0-Mini27.806.9020.9023.211.83
Qwen3.6-27B22.508.0014.5025.322.62
Gemma-4-31B-IT19.903.4016.5010.131.31
MiMo-V2.522.1012.709.4018.1411.01
InternVL3.5-8B16.706.909.8022.362.10

Multiple-choice questions follow the benchmark's scoring rule: an answer is counted as correct only if the model selects the correct answer under every tested option order. All evaluated models score substantially lower with text-only input than with video input, supporting the practical importance of visual evidence for answering Moment-Video questions. This trend holds for both closed-source and open-weight models, indicating consistent dependence on visual input across these evaluated systems. The difference between multiple-choice and open-ended questions also helps explain why some questions are still answered correctly without video. Multiple-choice questions directly expose candidate answers, allowing models to use language priors or eliminate implausible options. In addition, with the model temperature set to 0, models may more readily produce consistent answers across shuffled option orders, which is one reason for the higher multiple-choice scores. Open-ended questions instead require generating specific event counts, actions, or state changes. Their lower text-only accuracy is consistent with these details being difficult to recover from the question alone; remaining correct answers may arise from linguistic cues or guessing. These results have been added to Appendix G.6 of the revised manuscript.

Regarding the second request: Appendix E (Table 8) of the original paper reported a human solvability study in which participants answered questions after watching the corresponding videos. To provide a more detailed human reference, we further aggregated the original response records by the four subtasks and seven video domains and will add the corresponding results to Table 2. Each of the 15 retained participants answered 20 questions, yielding 300 responses to 200 sampled questions. Overall accuracy was 84.33%, and individual participant accuracies ranged from 70.00% to 95.00%. The following tables also include Seed-2.0-Pro, the best-performing video model overall in Table 2, as an overall reference. All values are accuracies (%).

EvaluationTOTCADTROverall
Human reference81.0882.3988.3583.3384.33
Seed-2.0-Pro50.3731.1447.0842.3539.60
EvaluationAIGCGUINatureIndustryGamesHuman activitiesAnimal
Human reference66.6782.8186.9692.5066.6792.2193.33
Seed-2.0-Pro45.5733.9469.4441.3239.3727.2055.00

These breakdowns show that, in the original sampled study, humans achieved accuracy above 80% on all four subtasks and answered most questions correctly across the seven video domains. This provides further evidence that the sampled questions are solvable when the videos are available. Compared with the human reference, video models still have substantial room for improvement on the full benchmark. The human baseline is now included in Table 2, and the related details have been added to Appendix E of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer rCWD part-2

W2:Seed-2.0 models use the native API with default settings, allowing the models to determine which frames to sample and how many, whereas most other evaluated models use 1 frame per second (1 FPS). Since the sampling rate strongly affects performance, Seed-2.0's advantage may partly arise from its input configuration. Could the authors report the actual number of frames used by Seed-2.0 and add results under a common 1 FPS setting for a fair comparison?

A2:Thank you for this valuable comment and for highlighting the need to evaluate Seed-2.0 under a common sampling configuration. The original experiments used the provider's default native video interface for the Seed series. Seed models are closed-source, and the existing evaluation logs do not provide the actual number of frames processed per video or the internal frame-selection rules. Therefore, the original results reflect the combined effects of each model and its provider's video preprocessing pipeline.

To address the request for a common 1 FPS setting, we reevaluated Seed-2.0-Pro, Lite, and Mini on all 1,000 questions. The client sampled at a target rate of 1 FPS with a maximum of 64 frames per video, matching the target sampling rate and frame cap used by most frame-sequence input models in the main experiments. When the number of candidate frames exceeded the budget, 64 frames were selected uniformly across the full video duration. The results are shown below:

ModelDefault native-video accuracy (%)Accuracy at 1 FPS, max. 64 frames (%)Change (percentage points)
Seed-2.0-Pro39.6024.00-15.60
Seed-2.0-Lite31.3033.10+1.80
Seed-2.0-Mini27.8026.30-1.50

The effects of input configuration differ across the three Seed models. Pro shows the largest decline, Lite improves slightly, and Mini changes relatively little. Under the common 1 FPS setting, Lite ranks first, followed by Mini and Pro, whereas Pro leads under the original default configuration. This shows that the Seed series is highly sensitive to the video input configuration. Therefore, Pro's leading performance in Table 2 should be interpreted in the context of its default native video processing pipeline. These results have been added to Appendix G.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer rCWD part-3

W3:Of the 1000 questions, Temporal Counting (TC) accounts for 472 (47.2%), whereas Temporal Reasoning (TR) contains only 85 (8.5%). Temporal Counting is also the most difficult task, so the overall metric is largely determined by counting performance. The paper describes the four tasks as assessing progressively higher capability levels, with the highest level involving reasoning about event-induced state changes. However, the only task at this level, Temporal Reasoning (TR), has the fewest samples. This imbalance may limit the benchmark's ability to measure temporal fidelity accurately.

A3:Thank you for pointing out that the imbalanced task distribution affects the overall score. Moment-Video evaluates video multimodal large language models at different levels, from perception to understanding and reasoning, including noticing brief events, understanding how they unfold, and inferring their consequences. Temporal Counting (TC) assesses a foundational perceptual capability: recognizing and tracking each occurrence of a brief event. Missing even one occurrence can change the answer, so counting itself remains challenging. Temporal Reasoning (TR) builds on this perceptual capability and requires models to infer consequences by combining events with pre- and post-event states. As video multimodal large language models improve, we plan to expand Moment-Video with more temporal reasoning questions that require integrating brief visual events and inferring their consequences. TC contains 472 questions (47.2%), while TR contains 85 (8.5%), so TC has the largest influence on the original overall accuracy aggregated over questions. To address this concern, we additionally report macro-average accuracy, calculated as (TO + TC + AD + TR) / 4. Each task receives a weight of 25%, regardless of its sample count. The recomputed results are shown below:

Macro rankModelTO (%)TC (%)AD (%)TR (%)Overall (%)Macro (%)
1Seed-2.0-Pro50.3731.1447.0842.3539.6042.74
2Seed-2.0-Lite34.8125.6436.0440.0031.3034.12
3Gemini-3-Flash41.4818.4331.8232.9426.9031.17
4Gemini-3.1-Pro32.5919.7033.4434.1226.9029.96
5Seed-2.0-Mini32.5922.8833.7725.8827.8028.78
6Gemini-3.1-Flash-Lite34.0718.6430.5227.0625.1027.57
7Kimi-2.625.9315.8936.6930.5924.9027.28
8DeepSeek-v4.1-Flash30.3714.6228.9031.7622.6026.41
9Qwen3.5-27B20.7419.0732.4727.0624.1024.84
10Qwen3.5-397B-A17B27.4114.8332.7923.5322.8024.64
11GPT-5.421.4812.0830.1929.4120.4023.29
12Qwen3.6-27B19.2617.1631.8223.5322.5022.94
13Gemma-4-31B30.3713.5624.3522.3519.9022.66
14Qwen3.5-122B-A10B25.1914.1927.6023.5320.6022.63
15MIMO-v2.518.5217.1630.8423.5322.1022.51
16Qwen3.6-35B-A3B17.0414.4129.2224.7120.2021.35
17Qwen3.5-35B-A3B20.0016.1026.3018.8220.0020.31
18Qwen3.5-9B18.5213.3526.6218.8218.6019.33
19Qwen3-VL-235B-A22B15.5612.0823.3823.5317.0018.64
20Qwen3.5-4B19.2613.7721.4317.6517.2018.03
20Qwen3-VL-30B-A3B18.5213.1420.4520.0016.7018.03
22InternVL3.5-241B-A28B19.2611.0220.4518.8215.7017.39
23InternVL3.5-8B15.5615.0420.1315.2916.7016.51
24GLM-4.6V15.568.6919.8121.1814.1016.31
25LLaVA-Video-72B13.3315.0418.8317.6516.2016.21
26Qwen3-VL-4B14.818.6919.4821.1813.9016.04
27Keye-VL-1.5-8B14.8112.9220.4514.1215.6015.58
28InternVL3.5-30B-A3B12.5916.1021.7511.7617.0015.55
29Qwen3-VL-8B15.5610.5917.8616.4714.0015.12
30InternVL3.5-4B14.8113.1417.867.0614.3013.22
31GLM-4.6V-Flash11.117.2014.2920.0011.0013.15
32VITA-1.519.268.4913.4210.5910.6012.94
33LLaVA-Video-7B11.1113.1413.967.0612.6011.32
34VideoLLaMA3-7B7.5213.6911.728.3311.8010.32

With equal weighting of the four tasks, Seed-2.0-Pro remains first with a macro-average accuracy of 42.74%. DeepSeek-v4.1-Flash has lower original overall accuracy than Qwen3.5-27B (22.60% versus 24.10%), but higher macro-average accuracy (26.41% versus 24.84%). These results have been added to Appendix G.7 of the revised manuscript.

Dear Reviewer rCWD,

Thank you very much for your time and effort in reviewing the submissions. As the authors’ responses have been available for several days, we would greatly appreciate it if you could assess whether your concerns have been adequately addressed and update your reviews and scores.

Reviewer_ojQyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Moment-Video, a benchmark designed to evaluate the ability of Video MLLMs to capture and reason about brief, answer-critical visual events. Through extensive model evaluations and diagnostic analyses, the authors reveal significant limitations in temporal fidelity, highlighting the challenges of understanding momentary visual events.

Critical strengths

1. Well-Motivated and Important Problem Definition: The paper highlights the limitations of existing Video MLLM benchmarks and focuses on brief, answer-critical visual events that are often overlooked in conventional evaluations. This is an important and meaningful problem, and the authors provide a clear motivation and well-defined evaluation setting. In particular, framing the difficulty of capturing such transient events through the concept of Temporal Fidelity is a valuable contribution.

2. Extensive Experimental Evaluation: The paper provides extensive experiments across 34 proprietary and open-source Video MLLMs, covering seven diverse domains and four task types. In addition to the main benchmark results, the authors conduct analyses of frame sampling rates, event coverage, and video duration, offering useful insights into the limitations of current Video MLLMs in understanding momentary visual events.

Critical weaknesses
  1. Insufficient Evidence That the Benchmark Isolates Temporal Fidelity: The authors define temporal fidelity as the ability to capture and reason over brief, answer-critical visual events. However, it remains unclear whether the proposed benchmark effectively distinguishes this capability from general visual perception and reasoning. For example, the low accuracy observed in Temporal Counting may result not only from failures to capture momentary events but also from limitations in general object tracking or counting. Additionally, some examples in Figure 3 appear to rely heavily on OCR or visual information extraction rather than temporal understanding. Controlled experiments, such as frame-order shuffling or removing answer-critical events, could help clarify whether the benchmark genuinely evaluates temporal fidelity as a distinct capability.

  2. Insufficient Details on the Selection and Verification of Answer-Critical Momentary Events: The paper constructs its benchmark around answer-critical momentary visual events, but the process of identifying and verifying such events is not sufficiently detailed. In particular, 842 of the 1,000 videos are collected from existing datasets and external sources, yet it is unclear how the authors determine whether a momentary event is truly essential to answering a question. Although the paper states that annotators first identify critical events, construct corresponding questions, and exclude samples that can be answered without observing those events, the specific criteria and verification procedures remain unclear. Further clarification is needed on how the authors ensure that answers cannot be inferred from surrounding visual context without observing the target events.

  3. Inconsistent Frame Budgets and Sampling Strategies Across Models: The evaluation employs different video input configurations across models, raising concerns about the fairness and interpretability of the performance comparisons. For instance, Seed-2.0-Pro is evaluated using its default video processing settings, while other models are evaluated with explicit frame sampling and limited frame budgets (64 or 128 frames). However, the actual number of frames processed by Seed-2.0-Pro is not reported, making it difficult to determine whether its superior performance stems from stronger temporal understanding or simply access to more visual information. Furthermore, differences between fixed-FPS sampling and model-specific video processing strategies may influence the results. A more controlled comparison under comparable frame budgets, along with an analysis of fixed versus adaptive sampling strategies, would help clarify the impact of input sampling on benchmark performance.

Questions for authors
  1. Evaluation of Answer-Critical Event Localization: Although the paper provides annotated event intervals for 300 videos, these annotations are only used for the event-coverage analysis. Given that identifying brief, answer-critical events is central to the proposed benchmark, could the authors additionally evaluate whether Video MLLMs can accurately localize these events? For example, evaluating the predicted event intervals against the ground-truth annotations using temporal grounding metrics could provide further insights into whether model failures arise from difficulties in locating critical events or understanding and reasoning about them.
5 comments
AuthorsOpenReview ↗

Response to Reviewer ojQy part-1

We sincerely thank you for the time and effort you devoted to reviewing our paper and for your constructive comments. In response, we have added ablation experiments on the capabilities assessed by the benchmark, clarified the data construction details, and added controlled sampling-rate analyses of closed-source models, active sampling experiments, and event localization experiments. We hope these revisions address your concerns. The changes have been incorporated into the revised manuscript and are detailed below.

W1:Insufficient Evidence That the Benchmark Isolates Temporal Fidelity.

A1:Thank you for this valuable comment and for suggesting additional validation through frame-order shuffling and critical-event removal. To examine whether performance on Moment-Video depends on temporal organization and answer-critical event evidence, we conducted two types of ablation across three models: shuffling frame order on all 1000 questions while retaining exactly the same visual frames, and removing frames corresponding to answer-critical events on a manually annotated subset of 300 questions. The results are shown in the table.

ModelTaskShuffling samplesOriginal → shuffled (change)Removal samplesOriginal → event removed (change)
Qwen3.6-27BTO13528.89→29.63 (+0.74)4017.50→5.00 (−12.50)
Qwen3.6-27BTC47221.40→11.02 (−10.38)14228.87→13.38 (−15.49)
Qwen3.6-27BAD30834.74→26.30 (−8.44)9238.04→23.91 (−14.13)
Qwen3.6-27BTR8530.59→24.71 (−5.88)2634.62→15.38 (−19.24)
Overall100027.3→19.40 (−7.90)30030.67→15.67 (−15.00)
InternVL3.5-30B-A3BTO13517.78→18.52 (+0.74)4017.50→7.50 (−10.00)
InternVL3.5-30B-A3BTC47214.19→10.81 (−3.38)14217.61→11.27 (−6.34)
InternVL3.5-30B-A3BAD30823.70→22.08 (−1.62)9225.00→17.39 (−7.61)
InternVL3.5-30B-A3BTR8516.47→11.76 (−4.71)2626.92→15.38 (−11.54)
Overall100017.80→15.40 (−2.40)30020.67→13.00 (−7.67)
Gemma-4-31BTO13532.59→34.07 (+1.48)4025.00→10.00 (−15.00)
Gemma-4-31BTC47217.16→8.05 (−9.11)14219.01→11.97 (−7.04)
Gemma-4-31BAD30831.17→28.57 (−2.60)9234.78→20.65 (−14.13)
Gemma-4-31BTR8527.06→23.53 (−3.53)2626.92→15.38 (−11.54)
Overall100024.40→19.20 (−5.20)30025.33→14.67 (−10.67)

First, the shuffling experiment shows task-dependent reliance on temporal organization. After shuffling, the overall accuracies of Qwen, Intern, and Gemma decrease by 7.90, 2.40, and 5.21 percentage points, respectively. Temporal Counting, Action Description, and Temporal Reasoning decline to varying degrees across all three models. This is consistent with these tasks requiring the ability to distinguish repeated event occurrences, integrate dynamic processes, or use temporal order. By contrast, Temporal Occurrence does not decline under shuffling, and Gemma improves by 1.48 percentage points. For questions asking whether an event occurs, retaining the critical observations is sufficient, and the original order is not necessary. Second, the critical-event removal experiment examines the models' dependence on brief answer-critical evidence. Removing critical events causes substantial declines across all four tasks. The response of Temporal Occurrence is particularly informative: performance remains broadly stable when critical observations are preserved but shuffled, yet declines when those observations are removed. This matches the task's dependence on the visibility of the critical event. Temporal Counting, Action Description, and Temporal Reasoning may additionally depend on the temporal organization of events and are therefore affected by both interventions. Together, the two interventions reveal different task dependencies on event visibility and temporal organization, supporting the benchmark's assessment of these capability dimensions. These results have been added to Section 4.6 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ojQy part-2

W2:Insufficient Details on the Selection and Verification of Answer-Critical Momentary Events.

A2:Thank you for this valuable comment and for pointing out that the screening and verification procedures for critical events require further explanation.

Data preparation. We apply the same screening criteria to all 1,000 videos, including the 842 videos from existing datasets and other public video sources. For each candidate clip, annotators first locate a brief, clearly visible action or state change, record its approximate timing, the entities involved, and the relevant pre- and post-event states, and then write the question and reference answer around this visual evidence.

Human verification. Two annotators cross-check whether each sample meets two conditions: the visual evidence in the video determines a unique answer, and the answer cannot be inferred solely from the surrounding visual context. Specifically, they check whether a viewer could still obtain the answer from pre- and post-event states or persistent visual cues even after missing the critical event. If such cues are found, the sample is revised or excluded. For persistent information that can be masked, the masked sample is retained only after passing another review under the same criteria. Gemini-3-Flash-Preview flags potential issues such as ambiguous wording, while human annotators make the final retention decisions.

Screening examples. Examples and the full procedure are provided in the benchmark construction pipeline (PDF). In the retained missile video, observing the intermediate yellow and orange stages is necessary to determine the complete sequence of color changes. In the gameplay video, the ammunition counter decreases from 30 to 27, allowing a viewer to infer that Genji, the game character, throws three shuriken even without observing the throwing moment. Therefore, the clip with the unmasked ammunition counter fails the initial screening. The figure illustrates masking the ammunition counter and reviewing the sample again.

These details have been added to Appendix D of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ojQy part-3

W3:Inconsistent Frame Budgets and Sampling Strategies Across Models.

A3:Thank you for this thoughtful question. You raise two concerns: the possible effects of closed-source models' frame budgets on their results, and the influence of sampling strategies on model performance.

Regarding the first concern: The original experiments used the provider's default native video interface for the Seed series. Seed models are closed-source, and their internal frame-selection rules and the actual number of frames processed per video cannot be recovered from the existing evaluation logs. To add a comparison with an explicitly controlled frame budget, we reevaluated Seed-2.0-Pro, Lite, and Mini on all 1000 questions. We sampled at 1 FPS with a maximum of 64 frames per video, matching the budget of the other open-source models in Table 2. The sampled images and their timestamps were provided in chronological order. The results are shown below:

ModelDefault native-video accuracy (%)Accuracy at 1 FPS, max. 64 frames (%)Change (percentage points)
Seed-2.0-Pro39.6024.00-15.60
Seed-2.0-Lite31.3033.10+1.80
Seed-2.0-Mini27.8026.30-1.50

The model ranking changes under the common configuration. With the shared frame budget, Lite ranks first, followed by Mini and Pro, whereas Pro leads under the original default configuration. Pro shows the largest decline, Lite improves slightly, and Mini changes relatively little, indicating that models differ in their sensitivity to video input configurations. Therefore, the leading results of Seed models and other closed-source native-video models under their default interfaces should be interpreted in the context of their native video processing pipelines. These results have been added to Appendix G.4 of the revised manuscript.

Regarding the second concern: Seed-2.0-Pro performs better with its default native-video configuration than with uniform sampling. To compare strategies, we reproduced and adapted AKS [1] and A.I.R. [2] using public code, alongside uniform sampling. AKS combines CLIP relevance with recursive temporal partitioning to balance question relevance and video coverage. A.I.R. uses coarse CLIP screening and vision-language feedback to revisit promising intervals and acquire new frames. We evaluated nine configurations on three models and 1,000 questions, with 64 chronologically ordered frames for final answering.

BackboneUniformAKS-CLIPA.I.R.-CLIP
Qwen3.6-27B21.4%21.2%20.5%
InternVL3.5-30B-A3B10.4%10.3%9.7%
Gemma-4-31B24.0%24.9%25.0%

AKS and A.I.R. improve Gemma by 0.9 and 1.0 percentage points, respectively, but slightly reduce Qwen and Intern's performance, showing no consistent gains. Trace analyses suggest different limitations.

For AKS, single-frame semantic relevance does not necessarily capture fine-grained temporal evidence. Recognizing a relevant object is not equivalent to distinguishing brief actions or repeated operations. In an audit of 300 questions with 609 necessary event intervals, AKS increases the number of events containing at least three distinct frames from 397 to 412, but reduces those containing at least one frame from 582 to 567. Denser sampling of selected events can miss other necessary evidence.

For A.I.R., additional observation does not consistently improve final evidence coverage. Traces confirm that it acquires frames beyond initial candidates and uses some for later analysis and answering. However, its feedback primarily measures single-frame relevance rather than whether observations resolve answer uncertainty. Its final inputs cover 559, 562, and 562 necessary intervals across the three models, respectively, below Uniform's 582.

Frame omission is not the only obstacle. On 43 questions whose source videos contain at most 64 frames, all strategies include every source frame, yet Qwen, Intern, and Gemma answer only 9, 5, and 13 correctly, respectively. Selection omissions cannot explain these errors.

Together, these results suggest that Moment-Video requires both locating relevant frames and preserving brief event-specific evidence. Existing frame-selection methods do not consistently resolve this challenge, motivating better active strategies. These results have been added to Section 4.5 of the revised manuscript.

[1]Tang X, Qiu J, Xie L, et al. Adaptive keyframe sampling for long video understanding[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 29118-29128.

[2]Zou Y, Jin S, Deng A, et al. Air: Enabling adaptive, iterative, and reasoning-based frame selection for video question answering[C]//International Conference on Learning Representations. 2026, 2026: 11302-11329.

Reviewer_rY1gOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Moment-Video, a benchmark for evaluating video MLLMs on brief, answer-critical visual events. It contains 1,000 human-verified video–question pairs across seven domains and four task types: temporal occurrence, temporal counting, action description, and temporal reasoning. The authors evaluate 34 models and investigate the effects of sampling rate, video duration, and event coverage. The benchmark addresses a meaningful evaluation gap. My main concern is that the diagnostic experiments do not sufficiently distinguish inadequate visual evidence from failures to interpret and reason over adequately observed events.

Critical strengths
  1. It gives a relevant and clearly motivated evaluation target. Understanding the overall content of a video does not necessarily require capturing brief events that determine an answer. This focus offers a useful perspective on video understanding beyond general scene recognition and video summarization.
  2. It provides an event-centered construction process with broad domain coverage. The inclusion of natural videos, GUI recordings, games, industrial activities, and generated content broadens the benchmark’s applicability.
  3. The frame-rate experiments and the event-coverage audit provide more information than overall accuracy alone. The paper also appropriately acknowledges that interval coverage does not guarantee sufficient dynamic evidence.
Critical weaknesses
  1. The full-coverage analysis does not establish that sufficient visual evidence was provided. In Appendix G.2, an event is considered covered if at least one input timestamp falls within its annotated interval. However, a single frame may be insufficient to resolve an action’s direction, temporal order, boundaries, or consequences.
  2. Nominal FPS does not reliably characterize the temporal information actually delivered to the model. For most explicit frame-sequence configurations, inputs are capped at 64 frames in the main setting and 128 frames in the higher-FPS experiments. Consequently, increasing the candidate sampling rate may produce little increase in effective input density for longer videos. For a 60-second video capped at 128 frames, both 8-FPS and 16-FPS candidate sampling ultimately provide roughly 2.1 frames per second on average.
  3. The evaluation does not systematically test active evidence acquisition. A model answering from a predetermined frame sequence faces a different problem from a system that can inspect a video coarsely, identify uncertain intervals, and revisit them with denser sampling. The paper does not report a controlled comparison between fixed sampling, question-conditioned frame selection, and iterative resampling. Such a comparison matters because poor performance may partly reflect the observation policy rather than the model’s ability to understand the relevant event once it is adequately observed.
Questions for authors

Overall, I think it could be a very useful benchmark and provide insightful findings. However, currently it lacks of in-depth research. I would like to raise my score if the weaknesses are addressed.

3 comments
AuthorsOpenReview ↗

Response to Reviewer rY1g part-1

We sincerely thank you for the time and effort you devoted to reviewing our paper and for your constructive comments. In response, we have reexamined the manually annotated data, analyzed the sampling-rate settings, and added active sampling experiments. We hope these revisions address your concerns. The changes have been incorporated into the revised manuscript and are detailed below.

W1:The full-coverage analysis does not establish that sufficient visual evidence was provided. In Appendix G.2, an event is considered covered if at least one input timestamp falls within its annotated interval. However, a single frame may be insufficient to resolve an action’s direction, temporal order, boundaries, or consequences.

A1:Thank you for this insightful comment. We agree that counting whether sampled frames fall within manually annotated event intervals, as in Appendix G.2, is insufficient to establish that the model receives enough visual information, because a single frame may omit relevant information. To address this concern, we extended the existing coverage analysis of 300 annotated videos with a human review of evidence sufficiency. Two annotators examined the sampled frames under the corresponding evaluation settings, together with the question and reference answer, to determine whether the visible evidence was sufficient to support an answer. The reference answer clarified which evidence needed to be checked, while the judgment remained grounded in the information actually visible in the sampled frames. The review confirmed that information was sometimes missing. The reasons included sampled timestamps falling within an interval without clearly showing the target event; incomplete coverage of the required event occurrences; and missing critical action dynamics or pre- and post-event states. We recomputed the statistics, and the updated results are shown below:

ModelFPSEvent hit (%)Full: n / Acc.Partial: n / Acc.Miss: n / Acc.Overall (%)
GPT-5.4141.765 / 38.5110 / 11.8125/ 23.222.3
GPT-5.4890.0253 / 35.631 / 12.916 / 25.032.7
Qwen3.5-27B144.369 / 34.8114 / 22.8117 / 23.926.0
Qwen3.5-27B897.4291 / 27.87 / 14.32 / 0.027.3
Qwen3.6-27B144.369 / 31.9114 / 23.7117 / 21.424.7
Qwen3.6-27B897.4291 / 31.67 / 0.02 / 0.030.7
Gemma-4-31B144.369 / 29.0114 / 14.0117 / 21.420.3
Gemma-4-31B894.3278 / 25.517 / 17.65 / 40.025.3
InternVL3.5-30B-A3B144.369 / 30.4114 / 18.4117 / 20.522.0
InternVL3.5-30B-A3B894.3278 / 21.217 / 11.85 / 20.020.7

As shown in the table, event evidence coverage increases from 41.7%–44.3% at 1 FPS to 90.0%–97.4% at 8 FPS. At 1 FPS, all five models achieve higher accuracy in the Full group than in the combined Partial and Miss groups. At 8 FPS, 278–291 samples are judged Full for the four open-weight models, yet their accuracy in this group remains only 21.2%–31.6%. This provides more reliable evidence that models still make many errors even when humans judge the sampled frames to contain sufficient evidence, and that missing critical evidence cannot explain all model failures. These updates have been incorporated into Appendix G.2 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer rY1g part-2

W2:The nominal frame rate does not reliably describe the temporal information actually delivered to the model. For most explicit frame-sequence configurations, the input is capped at 64 frames in the main setting and 128 frames in the higher-frame-rate experiments. Consequently, increasing the candidate sampling rate may do little to improve the effective input density for longer videos. For a 60-second video with a 128-frame cap, both 8 FPS and 16 FPS candidate sampling yield an average input density of approximately 2.1 frames per second.

A2:Thank you for this comment. The stated sampling rate is not necessarily equivalent to the actual sampling density, as we also explained in the original manuscript. The frame cap primarily prevents visual inputs from exceeding the model's available context budget when all candidate frames sampled at the specified rate are retained for a long video. Similar frame limits are common in existing video benchmarks. Video-MME-v2 [1] uses a maximum sampling configuration of 64 frames for some open-source models, depending on their long-context capabilities. LongVideoBench [2]compares maximum input budgets including 64 and 128 frames and reports that some models exceed their context length with 64 frames. The MiniCPM-V evaluation configuration in the official TempCompass[3] repository samples at approximately 1 FPS and uniformly subsamples to 64 frames when the number of candidate frames exceeds 64.

Furthermore, to examine the effect of increasing the sampling rate from 1 FPS to 8 FPS more directly, we reused the existing model predictions and conducted paired analyses on short-video subsets where the frame budget does not reduce the number of candidate frames. The results are shown below:

ModelPaired SamplesAccuracy at 1 FPS (%)Accuracy at 8 FPS (%)Change (%)
GPT-5.429917.0631.77+14.72
Qwen3.5-27B20024.5025.00+0.50
Qwen3.6-27B20024.5032.00+7.50
Gemma-4-31B20019.0025.00+6.00
InternVL3.5-30B-A3B20021.5019.00−2.50

As shown in the table, GPT-5.4, Qwen3.6-27B, and Gemma-4-31B improve substantially on the corresponding subsets. Denser sampling increases the chance of observing brief critical events and provides more evidence from adjacent moments, allowing action dynamics, repeated events, and pre- and post-event state changes to be represented more completely. However, increasing the sampling rate does not guarantee consistent gains for every model. Qwen3.5-27B gains little, while InternVL3.5-30B-A3B even declines. One possible explanation is that some models have limited long-context capabilities, so a large amount of contextual information may dilute the contribution of critical frames. These results have been added to Appendix G.3 of the revised manuscript.

[1]Fu C, Yuan H, Dong Y, et al. Video-MME-v2: Towards the next stage in benchmarks for comprehensive video understanding[J]. arXiv preprint arXiv:2604.05015, 2026.

[2]Wu H, Li D, Chen B, et al. Longvideobench: A benchmark for long-context interleaved video-language understanding[J]. Advances in Neural Information Processing Systems, 2024, 37: 28828-28857.

[3]Liu Y, Li S, Liu Y, et al. Tempcompass: Do video llms really understand videos?[C]//Findings of the Association for Computational Linguistics: ACL 2024. 2024: 8731-8772.

AuthorsOpenReview ↗

Response to Reviewer rY1g part-3

W3:The evaluation does not systematically examine active information acquisition. Models that answer from fixed frame sequences operate under a very different task setting from systems that can first scan a video coarsely, locate uncertain intervals, and then resample those intervals densely. The paper does not compare fixed sampling, question-conditioned frame selection, and iterative resampling. Such comparisons are essential: poor performance may result from the sampling and observation strategy rather than an inability to understand events once sufficient visual evidence is available.

A3:Thank you for this thoughtful question. Active frame selection and iterative observation have advanced video understanding, but the original manuscript focused on fixed sampling without systematically comparing information acquisition strategies. We therefore reproduced and adapted two methods using their public code: AKS [1] and A.I.R. [2]. This adds comparisons among fixed uniform sampling, question-conditioned frame selection, and iterative resampling.

AKS first computes CLIP relevance scores between candidate frames and the question, then recursively partitions temporal intervals and allocates the sampling budget to balance question relevance and coverage of the full video. AIR first performs coarse CLIP-based selection, then uses relevance feedback from a vision-language model to iteratively select promising intervals, acquire new frames, and increase local sampling density. We evaluated nine configurations across three models on the same 1,000 questions. All three methods use 64 sampled frames in chronological order for final answering. Results are shown below:

BackboneUniformAKS-CLIPA.I.R.-CLIP
Qwen3.6-27B21.4%21.2%20.5%
InternVL3.5-30B-A3B10.4%10.3%9.7%
Gemma-4-31B24.0%24.9%25.0%

Both methods improve Gemma by 0.9 and 1.0 percentage points, respectively, but slightly reduce performance on Qwen and Intern, showing no consistent gains. Trace and coverage analyses suggest different limitations. For AKS, the main limitation is the gap between single-frame semantic relevance and the need for fine-grained temporal evidence. Recognizing a question-relevant object does not imply recognizing a brief action or distinguishing similar operations. In an audit of 300 questions with positive events and 609 necessary event intervals, AKS increases the number of events containing at least three distinct frames from 397 under Uniform to 412, but reduces the number containing at least one frame from 582 to 567. This suggests that it can sample some relevant events more densely while missing other necessary events; improved local relevance may not compensate for the loss of complete event coverage. For AIR, the main limitation is that feedback-driven additional observation has not consistently translated into effective final evidence. Traces confirm that AIR acquires frames beyond the initial candidates and uses some for subsequent analysis and final answering. However, its feedback still primarily relies on single-frame relevance and does not directly measure whether the additional observations resolve uncertainty about the answer. On the same subset, final AIR inputs across the three models cover 559, 562, and 562 necessary intervals, respectively, all below Uniform's 582. This indicates that revisiting some intervals may still result in losing other events. Moreover, frame-selection omissions are not the only obstacle. For 43 questions whose source videos contain no more than 64 frames, all three strategies include every source frame, yet Qwen, Intern, and Gemma answer only 9, 5, and 13 questions correctly, respectively. These errors cannot be explained by frame omission.

Together, these results indicate that the capability emphasized by Moment-Video involves not only finding question-relevant frames, but also preserving the evidence needed to distinguish brief events. The two existing frame-selection methods do not yield substantial gains in this setting, supporting this focus. The results suggest that gains from relevant-frame selection in general video question answering cannot directly substitute for fine-grained understanding of momentary events, motivating more suitable active strategies. These results have been added to Section 4.5 of the revised manuscript.

[1]Tang X, Qiu J, Xie L, et al. Adaptive keyframe sampling for long video understanding[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 29118-29128.

[2]Zou Y, Jin S, Deng A, et al. Air: Enabling adaptive, iterative, and reasoning-based frame selection for video question answering[C]//International Conference on Learning Representations. 2026, 2026: 11302-11329.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.