- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces V-Horizon, a diagnostic benchmark for ultra-long video understanding. It contains 660 six-way multiple-choice questions derived from 34 complete television series ranging from 4 to more than 50 hours. The questions are divided into Temporal Tracking, Narrative Understanding, Visual Perception, and Visual Localization, with the first two intended to be relatively text-inferable and the latter two intended to require visual grounding. The paper evaluates 15 model and system configurations, including holistic VideoLLMs, agentic systems, and frame/token-selection baselines. It reports a large gap between performance on text-inferable and visually grounded questions. Oracle-evidence experiments suggest that locating relevant evidence is an important bottleneck. The paper also presents Agentic Skills, which searches subtitles and dense visual captions, retrieves clips, and optionally invokes a local VLM. Agentic Skills obtains 70.3% overall accuracy, compared with 55.3% for the strongest holistic baseline.
Critical strengths
- A useful diagnostic decomposition at a genuinely long temporal scale. Evaluating complete series of up to 50+ hours and separately reporting temporal, narrative, perceptual, and localization performance is more informative than another aggregate long-video QA score. The inclusion of localized evidence spans and oracle-evidence experiments makes the benchmark potentially valuable for studying where failures occur.
- The empirical results identify evidence retrieval as a concrete bottleneck. The large oracle gains on Visual Localization, together with the per-axis breakdown, provide useful evidence that uniform sampling and increased context alone do not reliably surface the required moments. This conclusion is consequential beyond the proposed Agentic Skills implementation.
Critical weaknesses
- Benchmark construction and the strongest evaluated system do not appear sufficiently independent. Candidate questions are generated using 10-second dense visual captions, while Agentic Skills searches a precomputed index containing dense visual captions from the same corpus. Most importantly, removing runtime clip retrieval and the VLM viewer reduces accuracy by only 0.7 points, from 70.3% to 69.6%. Thus, nearly all of the proposed system’s gain is retained when it reasons over textualized visual content rather than inspecting raw video at inference time. If the indexed captions are the same captions that helped generate candidate questions, the benchmark may preferentially reward information preserved by that representation. This substantially weakens the claim that the result demonstrates genuine long-range visual grounding.
- Dataset validation and comparison fairness are not yet strong enough for a benchmark paper. The paper does not report the number or qualifications of human annotators, independent answer verification, ambiguity rates, inter-annotator agreement, or adjudication procedures. The item-level rule used to assign questions to text-inferable versus visually grounded categories is also unclear. In addition, the systems in the main table use highly heterogeneous input and preprocessing budgets, while Agentic Skills benefits from exhaustive offline captioning and indexing. Reporting only end-to-end accuracy without matched preprocessing, token, latency, and monetary budgets makes the 15-point comparison difficult to interpret.
Questions for authors
- Are the dense captions searched by Agentic Skills the same caption instances, captioning model, and temporal segmentation used during question generation? Could the authors report performance on a human-authored subset, or on questions generated independently of the searchable captions, and rebuild the retrieval index using a different captioner or granularity?
- What was the complete human annotation and adjudication protocol? Please report the number of annotators, independent raw-video verification, inter-annotator agreement, ambiguity/removal rates, and the exact item-level criteria used to assign the four capability labels.
- How do the systems compare under matched total-resource budgets that include offline caption generation, video encoding, index construction, runtime model calls, visual tokens, latency, and monetary cost? It would also be helpful to evaluate calibration with one set of models and report final results on held-out models not involved in item filtering