acceptodds
Withdrawn from ICLR 2027

GEEZ: Grounded, Efficient, and Effective Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing video benchmarks can contain unanswerable or subjective questions, or questions solvable without the intended audio-visual evidence. We present GEEK, a benchmark of 1000 multiple-choice questions on web videos lasting 15 s. to 5 min.: 800 answerable from the video alone and 200 linking video evidence to verified external knowledge. We use a multimodal large language model to build EntityScript, a timestamped record of entities, events, and audio cues. An optimizer model iteratively refines prompts for generating questions from EntityScript, while web-search agents construct questions involving external knowledge. Candidate screening combines adversarial filtering with manual review. We also present GEEZ, a video question-answering framework that augments EntityScript with speech transcripts and on-screen text extracted offline. A planner prioritizes the transcripts for spoken wording and the extracted text for on-screen content, while querying an audio-visual recognizer for visual details. With Qwen3-Omni-30B as the recognizer, GEEX achieves 67.4% accuracy on GEEK. Compared with the EntityScript-based agent without these additional text sources (ES-AGENT), it improves accuracy by 8.5 percentage points while using 24% fewer query-time recognizer calls. It also outperforms a planner using only EntityScript, speech transcripts, and on-screen text by 17.9 points. Across recognizers, gains over ES-AGENT range from 4.1 to 5.1 percentage points on OmniVideoBench and Video-MME-v2. We will release the benchmark, construction records, video annotations, and framework code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Understanding, Multi-modal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WGNDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces GEEK, a 1,000-question video QA benchmark, and GEEZ, an agent that augments structured video descriptions with offline ASR and OCR tracks. GEEZ improves accuracy over the authors’ ES-AGENT baseline on GEEK and two external benchmarks.

Critical strengths

GEEZ improves over ES-AGENT across several recognizers and on two external benchmarks (Tables 2 and 4). The ASR/OCR ablations show that both tracks contribute to the gains (Table 3).

Critical weaknesses
  • Answer-key reliability remains unresolved. GEEK-ES-0401 includes “DAY 17” in its answer, but the supplement places it outside the queried interval. The authors flag this issue without updating the key or scores (Supplement §C, Fig. 5). Audit results for the remaining 497 GEEK-ES items are also missing.

  • AV-core screening does not establish that both modalities are necessary. Selection relies on one restricted-input model failing (§3.4). Two showcased questions already include the relevant speech in their stems (Supplement §C).

  • The ablations do not isolate the evidence-priority policy. They add ASR/OCR evidence without comparing alternative policies under fixed inputs and tools (Table 3). The gains therefore cannot be attributed to the priority rules.

  • Agent comparisons are limited to the authors’ ES-AGENT and ShotScript variants. No established agent is tested with matched inputs and budgets. No-Agent also receives a different query budget and only the first 600 seconds of audio on transfer benchmarks.

  • The efficiency claim needs qualification. At two questions per video, GEEZ and ES-AGENT both cost $41 per 1,000 questions (Supplement §F). GEEZ reduces query-time recognizer calls but adds offline processing. Reported latency excludes that processing.

Questions for authors
  • What is the verified key for GEEK-ES-0401? How do results change if it and other unresolved items are removed?

  • What fraction of AV-core items truly requires both modalities under independent human review?

  • What is the effect of the evidence-priority rules alone, with ASR/OCR tracks and tools fixed?

  • How does GEEZ compare with established video agents under matched inputs, compute, and total cost?

Reviewer_jYFuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that many VideoQA benchmarks can be solved merely by guessing options or relying on unimodal information. Therefore, this paper proposes GEEK, a comprehensive, evidence-grounded video understanding benchmark, and GEEZ, a video question answering framework. By leveraging structured video events, speech transcripts, on-screen text, and on-demand visual queries, GEEK enables more reliable evaluation of video understanding. Built upon the base framework ES-AGENT, GEEZ incorporates ASR, OCR and evidence source priority rules, yielding moderate improvements in accuracy and generalization.

Critical strengths
  1. This paper presents a more rigorous construction pipeline for video understanding benchmarks. Combining EntityScript, Prompt Optimization, Adversarial Filtering and human review, the authors aim to improve the answerability, evidence dependency and answer reliability of the questions.

  2. A multi-source evidence-based video question answering framework. GEEZ integrates structured events, speech transcripts, on-screen text and dynamic video queries to leverage the strengths of different evidence sources for reasoning.

  3. GEEZ can jointly improvements in accuracy and inference efficiency. Experiments show that GEEZ can boost accuracy across multiple recognizers and benchmarks, while reducing the number of invocations of the video recognition model during the query stage.

Critical weaknesses
  1. insufficient empirical support for the research motivation The authors argue that existing VideoQA benchmarks contain numerous questions that can be answered solely using textual cues, answer options, or unimodal information. For this reason, they propose the GEEK benchmark and adopt adversarial filtering to enhance evaluation quality. However, the paper fails to provide sufficient experimental evidence to quantify the prevalence and severity of such shortcut questions in mainstream existing benchmarks.

  2. Differences from OmniVideoBench Prior work such as OmniVideoBench has already adopted similar strategies including unimodal filtering, text-only filtering and human verification. Therefore, the substantive improvements of GEEK over these existing benchmarks in terms of evaluation quality remain unclear.

The ICLR 2026 published work OmniVideoBench itself includes:

  • Unimodal filtering to eliminate questions solvable using only audio or visual signals;
  • Text-only filtering to rule out questions answerable via commonsense knowledge or textual cues;
  • Human verification of answer correctness and uniqueness;
  • Stepwise annotation of multimodal evidence and reasoning chains.

As such, some construction principles proposed by GEEK are not novel. This raises a notable concern regarding GEEK’s research motivation and novelty: The authors need to clarify what new components GEEK adds to the filtering pipeline compared with OmniVideoBench, instead of merely stating that GEEK also uses adversarial filtering.

Admittedly, GEEK still has its own unique features, such as automated Prompt Evolution, Tutor-guided Web Agents, and 200 questions requiring external knowledge. These can constitute standalone contributions, but they should not be conflated with "being the first to address benchmark shortcuts".

  1. (minor weakness) The concept of Video Agent and Tool-augmented Video understanding in this paper is also not very new.
Questions for authors

Please refer to Critical Weaknesses.

The authors primarily need to address the following points:

  1. Sufficient quantitative experiments demonstrating that existing benchmarks indeed suffer from the aforementioned issues;
  2. The innovations of GEEK compared with OmniVideoBench;
  3. (optional) The innovations of GEEZ relative to prior video agents.
Reviewer_Fb1YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GEEK, a video question-answering benchmark, and GEEZ, a video understanding framework. GEEK contains 1,000 multiple-choice questions: 800 grounded in video content and 200 combining video evidence with external knowledge. Questions are constructed through EntityScript-based generation, followed by model-based screening under multiple input conditions, deduplication, and human review. GEEZ augments EntityScript with timestamped ASR transcripts and OCR text, using evidence-source priorities, text alignment, and local retrieval to support a planner. With Qwen3-Omni-30B as the recognizer, GEEZ achieves 67.4% accuracy on GEEK, outperforming ES-AGENT by 8.5 percentage points while reducing query-time recognizer calls by approximately 24%. The paper also reports improvements on OmniVideoBench and Video-MME-v2.

Critical strengths
  1. GEEZ outperforms ES-AGENT with all five recognizers, with gains of 7.4–8.5 percentage points on GEEK. Improvements of 4.1–5.1 points on two external benchmarks further support its practical value across evaluation settings. The simultaneous reduction in query-time recognizer calls suggests that the text tracks can eliminate some repeated perception overhead.
  2. The paper evaluates ASR alone, OCR alone, both tracks together, a planner-only configuration, and a single-call recognizer supplied with the text tracks. Full GEEZ outperforms these controls, suggesting that the benefits involve both additional textual evidence and subsequent evidence queries. Experiments with native-video input, an alternative ASR backbone, and changes to EntityScript extraction also reduce concerns that the results depend entirely on one specific configuration. The paper reports filtering counts, per-condition decision rules, answer distributions, and human audit results. It also correctly notes that a 0% text-only screening pass rate does not imply that all text-only models will fail. Disclosing incorrect keys, temporal-boundary issues, and statistically inconclusive results helps readers assess benchmark quality. The planned release of per-item screening records and construction histories would facilitate verification, although its completeness remains to be established.
Critical weaknesses

1 The AV-core definition does not convincingly verify the necessity of both audio and video modalities, and the handling of corrected annotation keys is underspecified. AV-core is identified by failure of one restricted-input model under audio-only and video-only settings, yet such failures can arise from model flaws, sampling noise or wrong ground truth rather than true audiovisual dependency. Some examples already contain transcribed speech and timestamps, further reducing the need for raw audio. Moreover, the paper uses corrected answer keys for evaluation but does not confirm whether filtering, consistency checks and AV-core membership labels were updated alongside these fixes; unresolved auditing issues leave the benchmark subset’s validity open to question.

2 Ablations fail to disentangle individual GEEZ components, and efficiency claims are limited to query-time rather than end-to-end gains. The experiments confirm the benefit of ASR/OCR text but cannot separate contributions from text alignment, source prioritization and local retrieval. Baseline variants alter model calling patterns and perception capacities and do not serve as clean controlled ablations. While GEEZ cuts online recognizer calls and frame processing, it introduces offline ASR/OCR overhead and higher planner token consumption. The reported latency and cost advantages rely on reusable preprocessing and do not demonstrate superior efficiency for single queries on unseen videos.

Questions for authors
  1. Beyond checking answer correctness, did the human audit assess whether each item could be solved using the question stem and only one modality? If such records exist, please report the proportion that actually requires both modalities, distinguishing cases where the stem already supplies the relevant audio information. This would directly affect my assessment of AV-core’s validity.
  2. Please clarify whether text-only screening, reference-model consistency, and AV-core membership were all reassessed against the final keys, and whether any items were removed or reclassified. Is the temporal-boundary example flagged in the appendix still included in the final evaluation? Please explain its final disposition and the impact of these corrections on the main results.
  3. Are there controlled results removing evidence-source priorities, disabling text alignment, or replacing local retrieval with direct provision of the same text, while keeping the planner, recognizer, query budget, and ASR/OCR content fixed? These results would help determine whether the gains primarily arise from more complete input information or from the proposed mechanisms for organizing and using evidence.
  4. What are the total cost and latency of GEEZ and ES-AGENT from video input to final answer in this setting? Please use a consistent accounting method that includes EntityScript construction and each method’s additional preprocessing, and clarify whether preprocessing steps run in parallel. This would establish the practical settings in which the efficiency advantage holds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.