- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces GEEK, a 1,000-question video QA benchmark, and GEEZ, an agent that augments structured video descriptions with offline ASR and OCR tracks. GEEZ improves accuracy over the authors’ ES-AGENT baseline on GEEK and two external benchmarks.
Critical strengths
GEEZ improves over ES-AGENT across several recognizers and on two external benchmarks (Tables 2 and 4). The ASR/OCR ablations show that both tracks contribute to the gains (Table 3).
Critical weaknesses
-
Answer-key reliability remains unresolved. GEEK-ES-0401 includes “DAY 17” in its answer, but the supplement places it outside the queried interval. The authors flag this issue without updating the key or scores (Supplement §C, Fig. 5). Audit results for the remaining 497 GEEK-ES items are also missing.
-
AV-core screening does not establish that both modalities are necessary. Selection relies on one restricted-input model failing (§3.4). Two showcased questions already include the relevant speech in their stems (Supplement §C).
-
The ablations do not isolate the evidence-priority policy. They add ASR/OCR evidence without comparing alternative policies under fixed inputs and tools (Table 3). The gains therefore cannot be attributed to the priority rules.
-
Agent comparisons are limited to the authors’ ES-AGENT and ShotScript variants. No established agent is tested with matched inputs and budgets. No-Agent also receives a different query budget and only the first 600 seconds of audio on transfer benchmarks.
-
The efficiency claim needs qualification. At two questions per video, GEEZ and ES-AGENT both cost $41 per 1,000 questions (Supplement §F). GEEZ reduces query-time recognizer calls but adds offline processing. Reported latency excludes that processing.
Questions for authors
-
What is the verified key for GEEK-ES-0401? How do results change if it and other unresolved items are removed?
-
What fraction of AV-core items truly requires both modalities under independent human review?
-
What is the effect of the evidence-priority rules alone, with ASR/OCR tracks and tools fixed?
-
How does GEEZ compare with established video agents under matched inputs, compute, and total cost?