- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces VideoZeroBench, designed to evaluate whether video multimodal large models can answer challenging long-video questions and accurately locate the spatiotemporal evidence supporting the answers. The dataset includes 138 videos and 500 open-ended questions, covering 13 domains and 11 types of capabilities, with an average video duration of approximately 667 seconds. Questions primarily involve fleeting details, small objects, counting, spatial relationships, and cross-clip evidence integration. The paper establishes a five-tier evaluation framework. Authors tested 19 models, achieving a top L3 accuracy of 24.8% and L5 of 1.8%. Independent human participants scored 67.6% and 55.8% on L3 and L5, respectively, across a subset of 50 questions. Experiments further demonstrate that directly providing correct time clips and spatial crops significantly improves question-answering performance, while tested multistep video reasoning methods have yet to yield notable joint localization benefits.
Critical strengths
- The research question has practical significance and focuses on the ability to overlook the accuracy of answers. The paper distinguishes between "answering correctly" and "being able to locate relevant evidence". For long video tasks, this distinction helps to discover differences between guessing, language priors, and real visual localization. Simultaneously reporting QA, time positioning, and spatial positioning is also more diagnostic value than relying solely on accuracy.
- The investment in data construction is sufficient, and the problem has clear challenges. The final question, answer, and evidence are constructed or verified manually and undergo two rounds of cross checking; The author also disclosed the filtering ratio of candidate questions and the annotation investment for approximately 200 people. The problem covers fine-grained visual clues and cross segment integration, supplementing the difficulties that are not easily exposed in conventional event recognition problems. The fixed output format also reduces reliance on LLM referees.
- The experiment covers a wide range and includes various useful robustness checks. In addition to the main rankings of different types of models, the paper also analyzed input modality, frame extraction budget, evidence clipping, inference rounds, and computational complexity during testing. The appendix further provides threshold sensitivity, joint and independent prediction, human performance, and repeated run results, indicating that the author considered multiple factors that may affect the conclusion.
- The evidence clipping experiment provides a clear direction for improvement. In Table 4, the accuracy of Gemini-3-Pro increased from 17.0% under full video input to 21.6% after temporal cropping, and further reached 41.2% after adding spatial cropping. This indicates that obtaining and presenting key evidence is an important bottleneck, providing empirical motivation for studying time search and spatial scrutiny.
Critical weaknesses
- The five level protocol can diagnose the ability, but it cannot directly verify the evidence fidelity of the answer. According to Section 2.3, the answers, time intervals, and spatial boxes are independently predicted under different prompts, and spatial prediction obtains true keyframe timestamps. Therefore, L5 actually measures the success of multiple tasks simultaneously, rather than the model actually using the output evidence in the same answer. The failure of positioning cannot directly prove that the previous correct answer has no visual basis.The joint prediction experiment in Appendix A.6 is helpful, but it only covers three models and 100 questions, and is not sufficient to fully validate this "evidence verification" explanation.
- Counting questions without annotations as zero points can confuse the meaning of cross level differences. Only 442 out of 500 questions have time annotations, while 320 questions also have time and space annotations, but L4/L5 all use 500 as the denominator. Therefore, even if all assessable questions are predicted correctly, the upper limits of L4/L5 are only 88.4% and 64.0%, respectively.The author has transparently disclosed this issue in the appendix and reported that the best model achieved scores of 16.1% and 2.8% on the corresponding subsets, therefore the conclusion that the model performs poorly still holds true. However, when directly comparing the decline in L3, L4, and L5 in the main text, it is not possible to fully explain the decline as insufficient localization ability. A more reasonable main result should be reported on a common annotated subset for each level of performance.
- The addition of new contributions to existing work requires more comprehensive empirical support. V-STaR has already decomposed video understanding and evaluated spatiotemporal reasoning around what, when, and where, so 'simultaneously examining answers and spatiotemporal localization' is not a completely new direction in itself. The clearer contribution of arxiv.org in this article lies in the combination of long videos, difficult details, manual evidence annotation, and five level protocols. These contributions are valuable, but currently the comparison with existing benchmarks is mainly at the attribute and case level. If it can be proved under the unified model and input budget that the new benchmark does reveal the wrong type or capability difference that the existing benchmark cannot distinguish, the contribution will be more convincing.
Questions for authors
The same as weaknesses.
5 comments
Rebuttal by Authors (2/2)
W3: About the Contribution Beyond V-STaR
The paper already provides a detailed comparison with V-STaR. Section 1 discusses differences from spatio-temporal grounding benchmarks, and Table 1 compares video duration, QA format, annotation sources, and hierarchical evaluation. Table 9 in Appendix A.3 further compares evaluation goals, data sources, and task characteristics.
V-STaR focuses on relatively salient object-event relations. As Figure 5 shows, many questions concern salient targets and explicit relations, such as “a dog behind a person.” The evidence in these examples is usually easier to recognize once the relevant frames are seen. VideoZeroBench instead requires identifying small cues in fleeting moments, such as a jersey number during a brief tackle, and combining them with semantic reasoning. These comparisons highlight differences in both temporal search range and the perceptual difficulty of the evidence. Together, they clarify our contribution: diagnosing temporal search, spatial localization, and complex reasoning in longer videos with finer-grained evidence.
We further support this comparison with data statistics and additional V-STaR results.
Quantitative comparison of data statistics.
| Statistic | VideoZeroBench | V-STaR |
|---|---|---|
| Mean video duration | 667.1 s | 110.23 s |
| Mean annotated box area / frame area | 6.8% | 19.77% |
These statistics quantify VideoZeroBench's longer temporal search range and smaller spatial targets.
Cross-benchmark results for Qwen3-VL-4B. We add V-STaR results using 1 FPS and a 96-frame cap:
| Benchmark | QA accuracy | Mean tIoU |
|---|---|---|
| V-STaR (Chain 1) | 37.3% | 24.9% |
| VideoZeroBench | 8.8% | 14.4% |
The same model scores substantially lower on VideoZeroBench in both QA and temporal localization.
Compared with V-STaR, VideoZeroBench emphasizes small targets, fleeting cues, and complex reasoning in longer videos. Its five-level protocol further distinguishes temporal search, spatial localization, and answer reasoning, revealing specific bottlenecks in challenging video understanding.
We hope these responses can address your concerns. We are committed to incorporating all suggested improvements in the revised version and welcome any further feedback.
Further Discussion or Feedback
Dear Reviewer V6fu,
As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.
Thank you again for your time and consideration!
Rebuttal by Authors (1/2)
Thank you for recognizing the importance of distinguishing answer correctness from evidence localization. We also appreciate your recognition of our manual annotations, two rounds of cross-verification, and ablation studies. Below, we provide detailed responses to the issues you raise.
W1: About the Evaluation Protocol
(1) Why evaluate the tasks separately? Joint prediction requires the model to answer, search for temporal intervals, and generate spatial boxes in one response. Its performance is therefore also affected by multi-task generation and compliance with a multi-field format. Separate prompts let the model focus on one task at a time. For spatial evaluation, we provide key frames and timestamps. Predicted boxes can then be compared with sparse manual annotations at the same times, separating spatial localization from frame selection. Section 3.1 and Appendices A.1 and A.6 explain these settings and their diagnostic purpose.
(2) Correct answers do not imply accurate evidence localization. On questions with both temporal and spatial annotations, only 9 of Gemini-3.7-Flash's 74 correct answers also pass both localization tasks. And for GPT-5.6-Sol, this is 8 of 61. The success rates among correct answers are therefore 12.16% and 13.11%. Therefore, even when asked specifically to locate evidence, models often fail on questions they have answered correctly. This shows limitations in their ability to locate evidence for these questions.
(3) The same observation holds under joint prediction. To address whether low joint success is specific to separate prompts, Appendix A.6 compares both protocols on 100 questions. In joint prediction, the model outputs its answer and evidence together and selects key-frame timestamps itself. We add manual box annotations at those timestamps to evaluate its spatial predictions.
None of the three models improves on L4/L5 under joint prediction. Both protocols show low joint success, with joint prediction performing even worse. Thus, the finding does not depend on separate inference calls.
W2: About Annotation Coverage and Cross-Level Comparison
(1) Why do some questions lack temporal or spatial annotations? The annotations depend on the evidence each question requires. As stated in §2.1, we annotate temporal intervals for questions tied to one or more distinct segments. We annotate spatial boxes when questions depend on specific small objects or regions in key frames. Not every question involves both forms of local evidence. Appendix A.1 details the annotation coverage.
(2) Why use 500 as the denominator? This keeps all five scores on a fixed question pool. L4 and L5 successively add temporal and spatial requirements to L3 correctness. A common denominator preserves L3 ≥ L4 ≥ L5. Shrinking denominators may otherwise yield higher percentages despite fewer successes. Each percentage point also represents five questions meeting that level's requirements. Section 2.3 and Appendix A.1 explain this convention. Appendix A.1 reports the coverage ceilings and the best model's scores on the corresponding annotated subsets.
(3) Comparison on the same annotated subset. To compare levels directly, we reuse six models' predictions on the same 320 questions with temporal and spatial annotations. We recompute L3–L5 using the paper's scoring rules:
All models still show a clear L3→L4→L5 decline with the same questions and denominator. The gap between accurate answers and precise spatio-temporal localization is therefore not solely due to annotation coverage.