acceptodds
Under review as a conference paper at ICLR 2027

VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification

18 views

Abstract

Video multimodal large language models achieve strong results on existing benchmarks, but answer accuracy alone does not establish whether they can locate the evidence needed to answer a question. We introduce VideoZeroBench, a challenging long-video benchmark with manually annotated question–answer pairs spanning 13 video domains. Questions target fine-grained cues, fleeting events, and evidence distributed across multiple segments. Temporal intervals and key-frame boxes are annotated where applicable. All questions undergo two rounds of crossverification for answer validity and evidence quality. Our five-level diagnostic protocol compares answering with and without evidence hints, then combines answer correctness with independently evaluated temporal and spatial grounding. Across 19 evaluated models, the best standard QA accuracy is 24.8% (Level-3), achieved by Gemini-3.7-Flash. No model exceeds 1.8% when correct answers and accurate spatio-temporal localization are jointly required (Level-5). Analyses of atomic abilities, evidence spans, input modalities, and thinking-with-videos inference further characterize where the evaluated systems struggle. These findings motivate more precise evidence search and localization for long-video question answering. Our code and data will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
Video Understanding Benchmark, Thinking with Videos

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_V6fuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VideoZeroBench, designed to evaluate whether video multimodal large models can answer challenging long-video questions and accurately locate the spatiotemporal evidence supporting the answers. The dataset includes 138 videos and 500 open-ended questions, covering 13 domains and 11 types of capabilities, with an average video duration of approximately 667 seconds. Questions primarily involve fleeting details, small objects, counting, spatial relationships, and cross-clip evidence integration. The paper establishes a five-tier evaluation framework. Authors tested 19 models, achieving a top L3 accuracy of 24.8% and L5 of 1.8%. Independent human participants scored 67.6% and 55.8% on L3 and L5, respectively, across a subset of 50 questions. Experiments further demonstrate that directly providing correct time clips and spatial crops significantly improves question-answering performance, while tested multistep video reasoning methods have yet to yield notable joint localization benefits.

Critical strengths
  1. The research question has practical significance and focuses on the ability to overlook the accuracy of answers. The paper distinguishes between "answering correctly" and "being able to locate relevant evidence". For long video tasks, this distinction helps to discover differences between guessing, language priors, and real visual localization. Simultaneously reporting QA, time positioning, and spatial positioning is also more diagnostic value than relying solely on accuracy.
  2. The investment in data construction is sufficient, and the problem has clear challenges. The final question, answer, and evidence are constructed or verified manually and undergo two rounds of cross checking; The author also disclosed the filtering ratio of candidate questions and the annotation investment for approximately 200 people. The problem covers fine-grained visual clues and cross segment integration, supplementing the difficulties that are not easily exposed in conventional event recognition problems. The fixed output format also reduces reliance on LLM referees.
  3. The experiment covers a wide range and includes various useful robustness checks. In addition to the main rankings of different types of models, the paper also analyzed input modality, frame extraction budget, evidence clipping, inference rounds, and computational complexity during testing. The appendix further provides threshold sensitivity, joint and independent prediction, human performance, and repeated run results, indicating that the author considered multiple factors that may affect the conclusion.
  4. The evidence clipping experiment provides a clear direction for improvement. In Table 4, the accuracy of Gemini-3-Pro increased from 17.0% under full video input to 21.6% after temporal cropping, and further reached 41.2% after adding spatial cropping. This indicates that obtaining and presenting key evidence is an important bottleneck, providing empirical motivation for studying time search and spatial scrutiny.
Critical weaknesses
  1. The five level protocol can diagnose the ability, but it cannot directly verify the evidence fidelity of the answer. According to Section 2.3, the answers, time intervals, and spatial boxes are independently predicted under different prompts, and spatial prediction obtains true keyframe timestamps. Therefore, L5 actually measures the success of multiple tasks simultaneously, rather than the model actually using the output evidence in the same answer. The failure of positioning cannot directly prove that the previous correct answer has no visual basis.The joint prediction experiment in Appendix A.6 is helpful, but it only covers three models and 100 questions, and is not sufficient to fully validate this "evidence verification" explanation.
  2. Counting questions without annotations as zero points can confuse the meaning of cross level differences. Only 442 out of 500 questions have time annotations, while 320 questions also have time and space annotations, but L4/L5 all use 500 as the denominator. Therefore, even if all assessable questions are predicted correctly, the upper limits of L4/L5 are only 88.4% and 64.0%, respectively.The author has transparently disclosed this issue in the appendix and reported that the best model achieved scores of 16.1% and 2.8% on the corresponding subsets, therefore the conclusion that the model performs poorly still holds true. However, when directly comparing the decline in L3, L4, and L5 in the main text, it is not possible to fully explain the decline as insufficient localization ability. A more reasonable main result should be reported on a common annotated subset for each level of performance.
  3. The addition of new contributions to existing work requires more comprehensive empirical support. V-STaR has already decomposed video understanding and evaluated spatiotemporal reasoning around what, when, and where, so 'simultaneously examining answers and spatiotemporal localization' is not a completely new direction in itself. The clearer contribution of arxiv.org in this article lies in the combination of long videos, difficult details, manual evidence annotation, and five level protocols. These contributions are valuable, but currently the comparison with existing benchmarks is mainly at the attribute and case level. If it can be proved under the unified model and input budget that the new benchmark does reveal the wrong type or capability difference that the existing benchmark cannot distinguish, the contribution will be more convincing.
Questions for authors

The same as weaknesses.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thank you for recognizing the importance of distinguishing answer correctness from evidence localization. We also appreciate your recognition of our manual annotations, two rounds of cross-verification, and ablation studies. Below, we provide detailed responses to the issues you raise.

W1: About the Evaluation Protocol

(1) Why evaluate the tasks separately? Joint prediction requires the model to answer, search for temporal intervals, and generate spatial boxes in one response. Its performance is therefore also affected by multi-task generation and compliance with a multi-field format. Separate prompts let the model focus on one task at a time. For spatial evaluation, we provide key frames and timestamps. Predicted boxes can then be compared with sparse manual annotations at the same times, separating spatial localization from frame selection. Section 3.1 and Appendices A.1 and A.6 explain these settings and their diagnostic purpose.

(2) Correct answers do not imply accurate evidence localization. On questions with both temporal and spatial annotations, only 9 of Gemini-3.7-Flash's 74 correct answers also pass both localization tasks. And for GPT-5.6-Sol, this is 8 of 61. The success rates among correct answers are therefore 12.16% and 13.11%. Therefore, even when asked specifically to locate evidence, models often fail on questions they have answered correctly. This shows limitations in their ability to locate evidence for these questions.

(3) The same observation holds under joint prediction. To address whether low joint success is specific to separate prompts, Appendix A.6 compares both protocols on 100 questions. In joint prediction, the model outputs its answer and evidence together and selects key-frame timestamps itself. We add manual box annotations at those timestamps to evaluate its spatial predictions.

ModelL4 separate / jointL5 separate / joint
Gemini-3-Pro10.0% / 7.0%2.0% / 1.0%
Gemini-2.5-Pro9.0% / 7.0%2.0% / 0.0%
Qwen3-VL-4B3.0% / 2.0%0.0% / 0.0%

None of the three models improves on L4/L5 under joint prediction. Both protocols show low joint success, with joint prediction performing even worse. Thus, the finding does not depend on separate inference calls.

W2: About Annotation Coverage and Cross-Level Comparison

(1) Why do some questions lack temporal or spatial annotations? The annotations depend on the evidence each question requires. As stated in §2.1, we annotate temporal intervals for questions tied to one or more distinct segments. We annotate spatial boxes when questions depend on specific small objects or regions in key frames. Not every question involves both forms of local evidence. Appendix A.1 details the annotation coverage.

(2) Why use 500 as the denominator? This keeps all five scores on a fixed question pool. L4 and L5 successively add temporal and spatial requirements to L3 correctness. A common denominator preserves L3 ≥ L4 ≥ L5. Shrinking denominators may otherwise yield higher percentages despite fewer successes. Each percentage point also represents five questions meeting that level's requirements. Section 2.3 and Appendix A.1 explain this convention. Appendix A.1 reports the coverage ceilings and the best model's scores on the corresponding annotated subsets.

(3) Comparison on the same annotated subset. To compare levels directly, we reuse six models' predictions on the same 320 questions with temporal and spatial annotations. We recompute L3–L5 using the paper's scoring rules:

ModelL3 / 320L4 / 320L5 / 320
Gemini-3.7-Flash23.13%14.06%2.81%
GPT-5.6-Sol19.06%6.25%2.50%
Gemini-3-Pro15.31%8.13%1.56%
Qwen3-VL-235B-A22B9.06%4.38%0.31%
Qwen3-VL-8B8.44%0.94%0.31%
Qwen3-VL-4B7.19%1.56%0.00%

All models still show a clear L3→L4→L5 decline with the same questions and denominator. The gap between accurate answers and precise spatio-temporal localization is therefore not solely due to annotation coverage.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W3: About the Contribution Beyond V-STaR

The paper already provides a detailed comparison with V-STaR. Section 1 discusses differences from spatio-temporal grounding benchmarks, and Table 1 compares video duration, QA format, annotation sources, and hierarchical evaluation. Table 9 in Appendix A.3 further compares evaluation goals, data sources, and task characteristics.

V-STaR focuses on relatively salient object-event relations. As Figure 5 shows, many questions concern salient targets and explicit relations, such as “a dog behind a person.” The evidence in these examples is usually easier to recognize once the relevant frames are seen. VideoZeroBench instead requires identifying small cues in fleeting moments, such as a jersey number during a brief tackle, and combining them with semantic reasoning. These comparisons highlight differences in both temporal search range and the perceptual difficulty of the evidence. Together, they clarify our contribution: diagnosing temporal search, spatial localization, and complex reasoning in longer videos with finer-grained evidence.

We further support this comparison with data statistics and additional V-STaR results.

Quantitative comparison of data statistics.

StatisticVideoZeroBenchV-STaR
Mean video duration667.1 s110.23 s
Mean annotated box area / frame area6.8%19.77%

These statistics quantify VideoZeroBench's longer temporal search range and smaller spatial targets.

Cross-benchmark results for Qwen3-VL-4B. We add V-STaR results using 1 FPS and a 96-frame cap:

BenchmarkQA accuracyMean tIoU
V-STaR (Chain 1)37.3%24.9%
VideoZeroBench8.8%14.4%

The same model scores substantially lower on VideoZeroBench in both QA and temporal localization.

Compared with V-STaR, VideoZeroBench emphasizes small targets, fleeting cues, and complex reasoning in longer videos. Its five-level protocol further distinguishes temporal search, spatial localization, and answer reasoning, revealing specific bottlenecks in challenging video understanding.

We hope these responses can address your concerns. We are committed to incorporating all suggested improvements in the revised version and welcome any further feedback.

AuthorsOpenReview ↗

Further Discussion or Feedback

Dear Reviewer V6fu,

As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.

Thank you again for your time and consideration!

Reviewer_zKjiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that QA accuracy alone cannot tell whether a video MLLM actually located the evidence it answered from, and introduces VideoZeroBench to test both jointly.

The benchmark is 500 manually written open-ended questions over 138 long videos (avg. 667s, 13 domains, Chinese and English), targeting fleeting frames, small objects, and evidence spread across segments. Answers are constrained to numbers or short phrases so no LLM judge is needed. Temporal intervals are annotated for 442 questions and key-frame boxes for 372 (320 have both), with two-round cross-verification and 17.8% of candidates discarded.

The contribution is a five-level protocol over the same questions: L1–L3 measure QA with spatio-temporal hints, temporal hints only, and no hints; L4 and L5 additionally require tIoU > 0.3, and tIoU + vIoU > 0.3, from separately prompted grounding predictions.

Validation covers 19 models. Best Level-3 accuracy is 24.8% (Gemini-3.7-Flash) and no model exceeds 1.8% at Level-5, while humans reach 67.6% / 63.5% / 55.8% at L3–L5 on a 50-question subset. Supporting analyses cover atomic abilities, evidence-span length, input modality and frame budget, thinking-with-videos inference, threshold sensitivity, separate-vs-joint prediction, and five repeatedruns.

Critical strengths

S1. The human reference makes the headline numbers interpretable. Benchmarks claiming near-zero SOTA usually cannot separate "models fail" from "the questions are unfair." Here participants reach 67.6/ 63.5 / 55.8% at L3–L5 under the same thresholds, against 24.8 / 14.2 / 1.8% for the best model — turning an unfalsifiable "impossible benchmark" claim into a measured gap. This is why I read the low scores as a finding, not an artifact.

S2. The construct is clean and under-served. Fixing the question set while varying evidence availability (L1–L3), then requiring independent grounding (L4–L5), isolates answering from localization in a way answer-only benchmarks cannot. The 40.0% vs. 24.8% L1–L3 gap is exactly what an accuracy leaderboard hides.

S3. The authors stress-test their own protocol. Threshold sensitivity (still 3.4% at the loosest setting), re-normalized denominators (16.1 / 2.8%), separate-vs-joint prediction on a shared subset, andfive repeated runs with std — each a place the claim could have collapsed and did not.

S4. Annotation discipline is real, not asserted. 200 person-hours, two verification rounds by non-authoring annotators, 108 of 608 candidates (17.8%) discarded, and short-form answers that remove LLM-judge noise.

S5. The thinking-with-videos negative result is useful. Iterative zoom-in gains 1.0–1.8 pp of QA yet leaves L4/L5 at zero, and localize-then-answer degrades Qwen3-VL-4B 7.8% → 7.0% — concrete and actionable for a popular line of work.

Critical weaknesses

W1. Level-5 has no measurement resolution. The top score of 1.8% is 9 of 500 questions, and 11 of 19 models sit at exactly 0.0. Table 13's ±0.09 covers decoding noise only, so question-sampling error dominates and no pairwise comparison here is meaningful. Part of it is protocol, not capability: L4/L5 use N=500 though only 442/320 questions carry annotations, and the re-normalized 16.1% / 2.8% are not the abstract's numbers. L4/L5 also multiply Level-3 correctness by grounding from a separate prompt, so "correct answer, wrong evidence" is partly the product of two low marginals (24.8 × 42.9≈10.6 vs. 14.2). A benchmark needs discriminative power over time; this one records a floor it cannot measure movement away from.

W2. Input conditions are not comparable, yet the cross-model conclusion rests on them. Gemini gets native video with audio, everyone else 1-FPS frames capped at 96–384 — and Table 5 shows audio accuracy moving 7.4% → 25.9% from that channel alone, while audio perception is a scored ability. Grounding compounds it: several open-source models show 1.8–3.6% tIoU despite decent Level-1 scores, and §A.1 zeroes unparseable outputs without reporting parse-failure rates, so "cannot ground" and "wrong format" are indistinguishable. The headline ordering is thus confounded with input channel, frame budget, and formatting.

Questions for authors
  1. What are the parse-failure rates for temporal intervals and boxes, per model? For the open-source models at 1.8–3.6% tIoU, how much of that is unparseable output rather than wrong localization (§A.1 zeroes both)?
  2. Can you report an iso-input comparison — e.g. Gemini-3.7-Flash and GPT-5.6-Sol both on 1-FPS frames without audio at a matched cap? Does the proprietary-vs-open-source ordering survive it?
  3. At Level-5, with 9 successes for the best model, what is the question-sampling uncertainty (e.g. a bootstrap CI over the 320 annotated questions)? Which pairwise differences in Table 2 remain significant?
  4. How correlated are Level-3 correctness and standalone tIoU per question? If roughly independent, what does Level-4 add beyond the product of the two marginals already in the table?
  5. Gemini produced the reference captions and question insights, and annotators could query it live (A.4). Were questions selected for Gemini failure, and how do you rule out a bias in the model ranking?
  6. Why does the abstract use the N=500 figures (1.8%) rather than the annotated-subset values (2.8%, A.1), given that coverage alone caps Level-5 at 64.0%?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thank you for recognizing the value of our five-level diagnostic protocol, annotation quality control, and protocol robustness analyses. We also appreciate your positive assessment of the thinking-with-videos experiments. Below, we provide detailed responses to the issues you raise.

Q1&W2: Details About Parsing Failures

We report parsing failure rates due to formatting errors. The denominators are 442 questions with temporal annotations and 372 with spatial annotations:

ModelTemporal parsing failure rateSpatial parsing failure rate
Qwen3-VL-4B5.66%2.96%
Qwen2.5-VL-7B1.36%8.60%
InternVL3.5-4B4.75%16.94%
Video-R1-7B0.45%6.99%

Using only parseable temporal predictions, mean tIoU changes from 1.83% to 1.86% for Qwen2.5-VL-7B and from 3.31% to 3.47% for InternVL3.5-4B. These values remain close to the original means. Their low scores therefore mainly reflect mismatches between predicted intervals and evidence, rather than parsing failures.

Similarly, excluding spatial formatting errors changes their mean vIoU from 1.40% to 1.54% and from 1.17% to 1.40%, respectively. Spatial localization scores thus remain low after removing formatting failures.

Q2&W2: About Matched-Input Model Comparisons

As explained in §3.1 and Appendix A.1, input formats and frame budgets reflect each model's interface, frame/context limits, and GPU memory constraints. This lets us evaluate models under favorable input conditions, using the same questions and scoring rules. Table 2 lists the frame budgets.

To check how input conditions affect model comparisons, we add a matched-input evaluation of Gemini-3.7-Flash, GPT-5.6-Sol, and Qwen3-VL-4B. All receive the same external inputs: target 1 FPS, at most 50 frames, and image height 360 pixels. We use temperature=0, without audio or evidence hints. Alongside the full-set results, we also report a subset of 150 non-audio questions: Every annotated temporal interval in this subset contains at least one sampled frame.

ModelL3: all 500 questionsL3: covered non-audio subset (150)
Gemini-3.7-Flash91/500 (18.20%)31/150 (20.67%)
GPT-5.6-Sol92/500 (18.40%)36/150 (24.00%)
Qwen3-VL-4B35/500 (7.00%)13/150 (8.67%)

Under matched inputs, Gemini-3.7-Flash and GPT-5.6-Sol achieve similar L3 accuracy: 18.20% and 18.40%. Their gap is smaller than in Table 2, while both still outperform Qwen3-VL-4B. Thus, their advantage over this open-source model persists after matching the inputs.

The paper also examines input modality in Table 5. Gemini-3-Pro achieves 17.0% overall L3 accuracy with both 384-frame input and native video with audio. Its audio-category accuracy, however, rises from 7.4% to 25.9%. This ablation tests one model's sensitivity to input modality. Our new comparison extends the analysis to differences between models under matched inputs.

Q3&W1: About Level-5 Resolution and Sampling Uncertainty

The full five-level results reveal model differences even when L5 scores are similarly low. L1–L3 compare answering under different evidence hints; L4 adds temporal localization requirements, while tIoU/vIoU measure localization precision. L5 tests whether answering and both localization tasks succeed on the same question. Low L5 scores for frontier models and zeros for some open-source models reflect the difficulty of this joint task. The large gap from human performance highlights a direction for future model research.

For Level-5, to assess sampling uncertainty, we compare the three models below. We reuse their predictions on the same 320 questions with temporal and spatial annotations, and resample by video 3,000 times (bootstrap). The table below uses the 320 annotated questions as the denominator:

ModelL5 successes / spatio-temporal annotated questionsL5 accuracy95% sampling confidence interval
Gemini-3.7-Flash9/3202.81%[1.22%, 4.59%]
GPT-5.6-Sol8/3202.50%[0.62%, 5.13%]
Qwen3-VL-235B-A22B1/3200.31%[0.00%, 0.99%]

To identify differences with statistical support, we also compare these models using paired video bootstrap under the settings in Table 2. For both Gemini-3.7-Flash and GPT-5.6-Sol versus Qwen3-VL-235B-A22B, the 95% confidence intervals for the L5 differences lie above zero. Both L5 differences therefore have statistical support. The interval for Gemini versus GPT includes zero and does not show a significant difference. Meanwhile, all three models' success-rate intervals have upper bounds no higher than 5.13%. Thus, even after accounting for sampling uncertainty, models rarely combine correct answers with precise spatio-temporal localization.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q4&W1: About QA–TG Correlation and Joint Success

On the same 442 temporally annotated questions, Pearson correlations between L3 correctness and continuous tIoU are 0.064 for Gemini-3.7-Flash and 0.087 for GPT-5.6-Sol. These small sample correlations alone do not establish dependence or independence.

To compare scoring methods, we compute the following four scores on these questions, defining temporal success as tIoU > 0.3.

Model(a) QA × mean tIoU(b) Continuous joint score(c) QA × temporal pass rate(d) L4
Gemini-3.7-Flash10.40%11.24%14.95%16.06%
GPT-5.6-Sol4.29%5.23%5.88%7.24%

(a) combines separate means and cannot identify whether correct answers and accurate localization occur on the same questions. (b) preserves that pairing, but scores continuous overlap without requiring localization to pass. (c) incorporates the temporal pass criterion, but still estimates joint success under independence. (d) L4 directly counts questions with both a correct answer and sufficient temporal localization precision. It preserves pairing and enforces localization precision without assuming independence, directly matching our goal of evaluating joint answer-and-evidence success.

Q5: About Dataset Construction and Selection Bias

Auxiliary references and manual question design. As explained in §2.1 and Appendix A.4, Gemini-2.5-Pro generates captions and some insights. These help annotators quickly understand long videos. Annotators then watch the full videos, design the final questions themselves, and manually determine and verify answers and evidence. The references do not determine the final questions or labels.

Gemini queries and retention criteria. The interface also allows Gemini queries to help annotators understand frontier models' reasoning and capability boundaries on difficult questions. Such queries are optional, and whether Gemini answers correctly is not a criterion for retaining a question. Instead, filtering checks answerability, answer uniqueness, and evidence accuracy (§2.1; Appendix A.4). After two rounds of manual cross-verification, 500 of 608 candidate questions are retained.

Cross-model results. Table 2 provides a further reference: Gemini-3.7-Flash outperforms GPT-5.6-Sol on L3 (24.8% vs. 18.8%), while GPT scores higher on spatial vIoU (16.6% vs. 10.2%). These results show different strengths of models across capabilities.

Q6&W1: About Annotation Coverage and Evaluation Denominators

(1) Why use 500 as the denominator? This makes all five scores success rates over the same fixed question pool. L4 and L5 successively add temporal and spatial requirements to L3 correctness. With a fixed denominator, adding these requirements preserves L3 ≥ L4 ≥ L5. A smaller denominator may otherwise yield a higher score despite fewer successes. Each percentage point also corresponds directly to five questions meeting that level's requirements. The abstract follows this definition from §2.3 and the main table. Appendix A.1 explains annotation coverage and its ceilings, and reports results on annotated subsets.

(2) Does the conclusion hold with different denominators? The best model still has nine L5 successes: 9/500=1.8% overall, or 9/320≈2.8% on questions with both annotation types. Likewise, L4 is 71/500=14.2% overall, or 71/442≈16.1% on temporally annotated questions. Thus, joint answer-and-evidence success remains low even when only evaluable questions count toward the denominator.

For a direct comparison across levels, we also recompute all scores on the same 320 questions with temporal and spatial annotations:

ModelL1 / 320L2 / 320L3 / 320L4 / 320L5 / 320
Gemini-3.7-Flash35.31%26.88%23.13%14.06%2.81%
GPT-5.6-Sol27.81%16.25%19.06%6.25%2.50%

With the questions and denominator fixed, Gemini-3.7-Flash still drops from 23.13% at L3 to 14.06% at L4 and 2.81% at L5. GPT-5.6-Sol drops from 19.06% to 6.25% and 2.50%. A substantial gap remains between correct answers and joint success with precise spatio-temporal localization.

We hope these responses can address your concerns. We are committed to incorporating all suggested improvements in the revised version and welcome any further feedback.

AuthorsOpenReview ↗

Further Discussion or Feedback

Dear Reviewer zKji,

As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.

Thank you again for your time and consideration!

Reviewer_8YeXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

VideoZeroBench evaluates difficult long-video questions together with their temporal and spatial evidence. It contains 500 manually constructed questions from 138 videos. Five evaluation levels compare answering with different evidence hints and combine answer correctness with localization. Across 19 models, the best standard QA score is 24.8%, while the best joint answer-and-spatio-temporal score is 1.8%.

Critical strengths
  1. The evidence annotations are the main value of this benchmark. Questions require precise, short answers, and temporal intervals and spatial boxes are manually constructed and cross-checked. This makes it possible to study failures that an answer-only evaluation would miss. The human study on a shared subset also provides a useful check that the questions are answerable.

  2. The paper does more than report low scores. In Table 4, giving annotated temporal segments and spatial crops raises Gemini-3-Pro from 17.0% to 41.2%, with similar improvements for the other two tested models. Total visual tokens are controlled in this comparison. This is convincing evidence that access to, and inspection of, the relevant details matters. The modest QA gains without grounding gains in the VideoChat-R1.5 experiment are also informative, although that finding is specific to the tested setup.

Critical weaknesses
  1. The cross-model comparison mixes reasoning ability with evidence availability. Gemini receives full video with audio, while most other systems use sampled frames with different caps. On questions built around fleeting events and small details, a model may fail because the necessary evidence never entered its input. Table 5 and the frame-budget ablations are useful checks, but do not establish evidence visibility for each model and question. This concern mainly affects standard QA and temporal search; including designated key frames for the Level-5 spatial task is a helpful safeguard.

  2. The grounding scores also combine different kinds of error. A model can locate the right event but miss the required temporal precision: the paper’s own 0.04-second example gives a two-second interval containing the event only 2% tIoU (Figure 12). That is a legitimate failure of precise localization, but it is different from searching in the wrong scene. The near-zero conjunctive Level-5 scores make these distinctions harder to see. Standalone IoUs help, but do not separate these cases among correctly answered questions. The threshold sweep and joint-prediction comparison support the broad conclusion that grounding is difficult; a breakdown of wrong-event, boundary-precision, and spatial-localization errors would make the diagnostic claims more useful.

Questions for authors

Could you report performance on a subset where the required evidence is confirmed to be present in each model’s actual input, using a common budget where possible? Among temporal-grounding failures, how many identify the relevant event but use an overly broad interval? Reporting localization success conditional on a correct answer would also help interpret the gap between QA and joint success.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for valuing our manual evidence annotations and cross-verification. We also appreciate your recognition of our fixed-token evidence-cropping experiments, human study, and video reasoning agent analysis. Below, we provide detailed responses to the issues you raise.

W1&Q1: About Matched Inputs and Evidence Coverage

We add an L3 comparison with matched inputs and check temporal coverage from the actual frames. We test whether model differences persist and whether sampling the relevant intervals enables correct answers.

Matched inputs and coverage checks. Gemini-3.7-Flash, GPT-5.6-Sol, and Qwen3-VL-4B use the same input format: target 1 FPS, at most 50 frames, and image height 360 pixels. All use temperature=0, without audio or evidence hints. We select 150 non-audio questions: timestamps confirm that every annotated interval contains at least one sampled frame. Their L3 inputs thus include visual information from each annotated interval. Of these, 61 questions are answerable from a single frame.

ModelAll 500 questionsCovered subset (150)Single-frame-answerable subset (61)
Gemini-3.7-Flash91/500 (18.20%)31/150 (20.67%)15/61 (24.59%)
GPT-5.6-Sol92/500 (18.40%)36/150 (24.00%)14/61 (22.95%)
Qwen3-VL-4B35/500 (7.00%)13/150 (8.67%)3/61 (4.92%)

The gap between proprietary and open-source models persists under matched inputs. All three models improve slightly on the 150 covered questions, but accuracy remains at most 25%. Sparse sampling may miss motion cues. We therefore also evaluate the 61 questions in this subset that are answerable from a single frame. Performance remains low, showing limitations in locating evidence and reasoning from it.

We also compare input conditions in the paper: Evidence cropping. Table 4 compares full videos, annotated temporal segments, and segments with spatial crops at a fixed total visual token budget. Spatial evidence coverage. Standard L5 provides designated key frames and timestamps (§3.1; Appendices A.1 and A.6). It separates spatial localization from frame selection and tests localization with the corresponding frames available. Frame rate and budget. Appendix A.2, Table 10 tests Qwen3-VL-4B and Qwen3-VL-235B-A22B on L3. It compares 48/96/192/384-frame caps at 1 FPS and a 768-frame cap at 2 FPS. Input modality. Table 5 tests Gemini-3-Pro with no audiovisual input, audio only, sampled frames, and native video.

W2&Q2: About Localization Errors

We break down localization failures and their relationship to answer correctness, using the settings in Table 2.

(1) Types of temporal localization failures.

We group parseable outputs with tIoU ≤ 0.3 by overlap with the annotations. The table also separates those with correct L3 answers:

ModelScopeTemporal failuresZero overlapFull coverage but overly broadOther partial overlap
Gemini-3.7-FlashAll parseable temporal failures1694784/169 (49.70%)38
Gemini-3.7-FlashCases with correct L3 answers36421/36 (58.33%)11
GPT-5.6-SolAll parseable temporal failures26411968/264 (25.76%)77
GPT-5.6-SolCases with correct L3 answers441817/44 (38.64%)9

(2) Full coverage does not imply precise temporal localization.

Broad intervals can contain an event without meeting the required precision (Appendix A.1; Figure 12). For Gemini-3.7-Flash, predictions covering all evidence but failing the tIoU threshold span 8.03 times the annotated duration (median). In the group, L3 accuracy is 25.0%, and 75% of answers are wrong. Containment alone neither establishes precise localization of key temporal details nor guarantees correct answers.

(3) Localization failures on correctly answered questions.

Among correctly answered questions with temporal annotations, Gemini-3.7-Flash passes temporal localization (tIoU > 0.3) on 71/107 (66.36%). GPT-5.6-Sol passes on 32/82 (39.02%). On the same 320 questions with both annotation types, we further divide correct L3 answers by localization outcome:

ModelCorrect L3 answersTemporal localization failedTemporal passed, spatial failedBoth temporal and spatial passed
Gemini-3.7-Flash7429369
GPT-5.6-Sol6141128

Among correctly answered questions, 12.16% for Gemini and 13.11% for GPT also pass both localization tasks. This breakdown distinguishes temporal failures from spatial failures after successful temporal localization.

The paper also examines threshold sensitivity (Appendix A.2). For Gemini-3-Pro, lowering both tIoU and vIoU thresholds from 0.3 to 0.1 raises L5 from 1.0% to 3.4%. Joint success remains low even with looser thresholds.

We hope these responses can address your concerns. We are committed to incorporating all suggested improvements in the revised version and welcome any further feedback.

AuthorsOpenReview ↗

Further Discussion or Feedback

Dear Reviewer 8YeX,

As the discussion period is approaching its end, we would greatly appreciate any further feedback on our responses, especially regarding whether our clarifications have addressed your main concerns.

Thank you again for your time and consideration!

Thank you for the thorough response, which addresses my main concerns; I will maintain my positive score.

Dear Reviewer 8YeX:

We sincerely appreciate your thoughtful and constructive feedback. We are glad that our rebuttal addresses your concerns, and we appreciate your positive assessment of the paper.

Comments on the paper

1 comment
Area_Chair_bAHXOpenReview ↗

Please engage in the discussion

Dear Reviewers 8YeX and zKji,

Thank you for your reviews and contributions so far. The authors have now posted their response, and I would like to kindly remind you to review it and engage in the discussion where appropriate.

Please also consider whether the authors' clarifications address any of your concerns and update your assessment if warranted.

Thank you for your time and effort.

Best regards,

Your AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.