acceptodds
Withdrawn from ICLR 2027

V-Horizon: A Diagnostic Benchmark for Ultra-Long Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding videos that span tens of hours requires not only identifying sparse evidence but also interpreting it in context. We introduce V-Horizon, a diagnostic benchmark comprising 660 six-choice questions from 34 complete TV series, with durations ranging from 4 to over 50 hours. The benchmark evaluates four complementary dimensions: Temporal Tracking (T), Narrative Understanding (N), Visual Perception (V), and Visual Localization (W), separating questions that are largely answerable from textual information from those that require visual evidence. We evaluate 15 models and systems, including variants based on frame selection and visual-token compression. Controlled input comparisons show that subtitle-only settings yield substantially larger gains on T and N than on V and W, confirming the distinct dependence of the latter on visual information. Providing oracle evidence improves W accuracy by up to 38.6 percentage points, revealing a strong sensitivity to evidence accessibility. We further introduce Agentic Skills, a retrieval-based framework that searches subtitles and dense visual captions and can optionally invoke a local VLM for targeted inspection. Agentic Skills achieves 70.3% overall accuracy, compared with 55.3% for the strongest holistic configuration in our main comparison. Even without runtime visual tools, it retains 69.6% accuracy, indicating that indexed visual descriptions and targeted retrieval account for much of the performance gain. Overall, V-Horizon provides a controlled testbed for studying evidence access in long-form video understanding while helping distinguish capability limitations from failures to retrieve the information needed to answer a question.

Primary area:
datasets and benchmarks
Keywords:
benchmark, long video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oS2AOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces V-Horizon, a diagnostic benchmark for ultra-long video understanding. It contains 660 six-way multiple-choice questions derived from 34 complete television series ranging from 4 to more than 50 hours. The questions are divided into Temporal Tracking, Narrative Understanding, Visual Perception, and Visual Localization, with the first two intended to be relatively text-inferable and the latter two intended to require visual grounding. The paper evaluates 15 model and system configurations, including holistic VideoLLMs, agentic systems, and frame/token-selection baselines. It reports a large gap between performance on text-inferable and visually grounded questions. Oracle-evidence experiments suggest that locating relevant evidence is an important bottleneck. The paper also presents Agentic Skills, which searches subtitles and dense visual captions, retrieves clips, and optionally invokes a local VLM. Agentic Skills obtains 70.3% overall accuracy, compared with 55.3% for the strongest holistic baseline.

Critical strengths
  1. A useful diagnostic decomposition at a genuinely long temporal scale. Evaluating complete series of up to 50+ hours and separately reporting temporal, narrative, perceptual, and localization performance is more informative than another aggregate long-video QA score. The inclusion of localized evidence spans and oracle-evidence experiments makes the benchmark potentially valuable for studying where failures occur.
  2. The empirical results identify evidence retrieval as a concrete bottleneck. The large oracle gains on Visual Localization, together with the per-axis breakdown, provide useful evidence that uniform sampling and increased context alone do not reliably surface the required moments. This conclusion is consequential beyond the proposed Agentic Skills implementation.
Critical weaknesses
  1. Benchmark construction and the strongest evaluated system do not appear sufficiently independent. Candidate questions are generated using 10-second dense visual captions, while Agentic Skills searches a precomputed index containing dense visual captions from the same corpus. Most importantly, removing runtime clip retrieval and the VLM viewer reduces accuracy by only 0.7 points, from 70.3% to 69.6%. Thus, nearly all of the proposed system’s gain is retained when it reasons over textualized visual content rather than inspecting raw video at inference time. If the indexed captions are the same captions that helped generate candidate questions, the benchmark may preferentially reward information preserved by that representation. This substantially weakens the claim that the result demonstrates genuine long-range visual grounding.
  2. Dataset validation and comparison fairness are not yet strong enough for a benchmark paper. The paper does not report the number or qualifications of human annotators, independent answer verification, ambiguity rates, inter-annotator agreement, or adjudication procedures. The item-level rule used to assign questions to text-inferable versus visually grounded categories is also unclear. In addition, the systems in the main table use highly heterogeneous input and preprocessing budgets, while Agentic Skills benefits from exhaustive offline captioning and indexing. Reporting only end-to-end accuracy without matched preprocessing, token, latency, and monetary budgets makes the 15-point comparison difficult to interpret.
Questions for authors
  1. Are the dense captions searched by Agentic Skills the same caption instances, captioning model, and temporal segmentation used during question generation? Could the authors report performance on a human-authored subset, or on questions generated independently of the searchable captions, and rebuild the retrieval index using a different captioner or granularity?
  2. What was the complete human annotation and adjudication protocol? Please report the number of annotators, independent raw-video verification, inter-annotator agreement, ambiguity/removal rates, and the exact item-level criteria used to assign the four capability labels.
  3. How do the systems compare under matched total-resource budgets that include offline caption generation, video encoding, index construction, runtime model calls, visual tokens, latency, and monetary cost? It would also be helpful to evaluate calibration with one set of models and report final results on held-out models not involved in item filtering
Reviewer_kXbZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents V-Horizon, a benchmark containing 660 questions from 34 full-length TV series. It separates questions that can be answered from text from those that require visual evidence and finds a clear performance gap between the two. The proposed Agentic Skills system achieves 70.3% accuracy, 15 points higher than the best holistic baseline. These results point to evidence retrieval as a major bottleneck, though much of the improvement comes from searching pre-generated visual captions rather than directly examining the video.

Critical strengths
  1. The paper studies an important and relatively underexplored problem: understanding complete videos that span several to more than 50 hours. V-Horizon is broad, with 660 questions from 34 TV series across several genres. The separation into Temporal Tracking, Narrative Understanding, Visual Perception, and Visual Localization is useful. In particular, reporting text-inferable and visually grounded performance separately gives more insight than a single aggregate score.
  2. The evaluation covers a wide range of models and systems, including agentic methods, closed- and open-source VideoLLMs, and frame- and token-selection baselines. The oracle-retrieval experiments and tool ablations also provide useful evidence about the role of evidence retrieval in long-video understanding.
Critical weaknesses

My major concern is that the benchmark construction and Agentic Skills use the same intermediate representations. If I understand the construction pipeline correctly, candidate questions are discovered using ASR transcripts and dense visual captions, while Agentic Skills also searches over these representations when answering the questions. Although the final questions and answers are verified against the raw video, this setup may still favor the proposed retrieval pipeline. If my understanding is incorrect, the paper would benefit from clarifying these details. Otherwise, including a subset of manually written questions would provide a fairer evaluation.

Questions for authors
  1. Please see Critical Weaknesses.
  2. Given that Agentic Skills uses an offline preprocessing pipeline that includes ASR, dense visual captioning, and multimodal embedding and indexing, the paper should report the computational cost and running time of these steps, as well as the API usage of the full pipeline, to assess its practicality.
  3. The paper emphasizes cross-episode reasoning as a key feature of V-Horizon, but relavant details is not sufficiently presented. The paper does not provide clear examples of what these questions look like and report how many questions actually require information from multiple episodes. A comparison of how different models perform on these questions would also be useful.
  4. The methodological novelty of Agentic Skills appears limited. The framework primarily orchestrates existing components, including BM25/grep-based text retrieval, LanguageBind-based clip retrieval, an off-the-shelf LLM orchestrator, and a local Qwen3-VL viewer. It is therefore better viewed as a strong system integration. If my understanding is inaccurate, could you please clarify it?
Reviewer_5QAoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces V-Horizon, a benchmark for ultra-long video understanding consisting of 660 six-option multiple-choice questions drawn from 34 complete television series spanning 4 to over 50 hours.

The benchmark distinguishes four capabilities: Temporal Tracking (T), Narrative Understanding (N), Visual Perception (V), and Visual Localization (W). The first two are considered primarily text-inferable, while the latter two require visual evidence.

The authors evaluate 15 models and systems, finding that models generally perform substantially better on text-inferable questions than on visually grounded ones. Oracle experiments suggest that access to relevant evidence significantly improves performance, particularly on visual localization tasks.

The paper also introduces Agentic Skills, a tool-augmented system combining subtitle and visual-caption retrieval, multimodal clip retrieval, and localized VLM inspection. The system achieves 70.3% overall accuracy, compared with 55.3% for the strongest holistic baseline.

Critical strengths
  1. Relevant benchmark setting and a useful diagnostic perspective.

Evaluating understanding over complete television series is an interesting and challenging setting. I appreciate the authors' effort to separate questions that can be answered primarily from linguistic information from those that require visual grounding.

The controlled-input experiments in Appendix D are particularly useful. They demonstrate that performance on Temporal Tracking and Narrative Understanding improves substantially with subtitles, whereas Visual Perception and Visual Localization remain difficult under the subtitle-only setting.

  1. Interesting experimental evidence regarding evidence retrieval.

The oracle experiments demonstrate that access to relevant evidence can substantially improve performance. The evaluation also includes a reasonable variety of holistic, agentic, and frame-selection baselines.

The work highlights an important practical consideration: long-video understanding is not only a question of model capacity or context length, but also of how relevant information is located and represented.

Critical weaknesses
  1. The main narrative about agentic visual understanding is not fully supported by the ablations and comparisons.

The paper argues that its agentic framework improves performance by dynamically retrieving and inspecting relevant visual evidence. However, the results in Appendix F (Table 9) substantially qualify this interpretation.

Removing both runtime visual clip retrieval and the VLM viewer reduces accuracy from 70.3% to only 69.6%. On Visual Perception, accuracy changes from 60.4% to 59.5%, and on Visual Localization from 70.9% to 68.1%.

This suggests that almost all the reported performance of Agentic Skills can be obtained without invoking its runtime visual tools. The authors correctly acknowledge that the remaining index still contains visual information in the form of dense captions. Nevertheless, this changes the interpretation of the contribution: the main advantage may come from offline visual captioning and text retrieval, rather than dynamic visual inspection or the proposed tool-use mechanism.

This distinction is particularly important because the comparison against holistic models is not controlled for evidence accessibility. Agentic Skills operates over an index with captions generated every 10 seconds, while GPT-5.6 receives only 256 frames per series and Gemini receives up to 10 sampled video segments. These configurations expose very different quantities of information about the original video.

Although the authors acknowledge the unequal input budgets, the paper repeatedly uses the resulting performance differences to support claims about the superiority of agentic retrieval and visual reasoning. The experiments do not isolate these factors sufficiently.

A more convincing evaluation would compare Agentic Skills against a strong non-agentic text-retrieval baseline using exactly the same precomputed index, and ablate the contributions of subtitles, dense visual captions, iterative search, and runtime visual inspection separately.

I would also distinguish the conclusions more carefully: the results support the importance of retrieving information derived from videos, but do not establish that runtime visual inspection is an important source of the system's performance advantage.

  1. The benchmark's novelty and diagnostic validity need stronger justification.

The use of full-length television series is interesting, but closely related benchmark designs already exist.

In particular, SeriesBench (CVPR 2025) evaluates narrative understanding over 105 series; SagaQA (SIGDIAL 2026) evaluates multi-hop multimodal reasoning across complete television series; and InfiniBench (EMNLP 2025) studies grounding, reasoning, and linguistic shortcuts in long-form movies and television content. These works are highly relevant to the paper's motivation but are absent from its related-work comparison.

More fundamentally, the authors do not establish how many questions genuinely require long-horizon or cross-episode understanding. The benchmark includes examples of fine-grained visual recognition at a specified timestamp and localization of an individual event. Such questions can be answered from short local evidence once the appropriate moment is identified.

These are legitimate retrieval tasks, but they do not necessarily require understanding or integrating information across tens of hours. Simply associating a localized question with a very long source video does not establish a long-horizon reasoning requirement.

The authors should quantify how many questions require multiple evidence segments, how far apart these segments occur, and what fraction genuinely requires cross-episode reasoning. A breakdown of performance by evidence separation and series duration would make the benchmark's distinguishing contribution substantially more convincing.

The oracle experiments also deserve a more cautious interpretation. In particular, the large improvement in W accuracy (47.3% to 85.9% for Gemini 3.1 Pro) is difficult to interpret independently because W explicitly asks models to locate events in time. Supplying ground-truth evidence spans, potentially including timestamp information, may reveal much of the answer to the localization task. The paper should clarify exactly what information is exposed in the oracle setting and separate performance gains caused by revealing target timestamps from gains in visual comprehension.

Finally, the benchmark construction relies on automatic question generation, filtering, and verification, but does not provide sufficient independent human validation statistics for question correctness, ambiguity, or the claimed cross-episode and modality requirements.

Taken together, these issues make it difficult to assess how much new diagnostic information V-Horizon provides beyond existing long-video evaluation benchmarks.

Questions for authors
  1. What is the principal contribution of Agentic Skills beyond its precomputed visual-caption index?

Given Table 9, could the authors compare against a non-agentic retrieval baseline with the same index and answer model? An ablation removing dense visual captions while retaining subtitles would also help identify which representation contributes most to the final results.

  1. How many questions actually require long-horizon reasoning?

Could the authors provide the distribution of evidence-span lengths, temporal separation between evidence segments, and the proportion of questions that require combining information across different episodes? This is important for distinguishing long-horizon understanding from sparse-event retrieval within a long video.

  1. Does the oracle localization experiment expose the answer through timestamp information?

How exactly are the ground-truth evidence frames presented to models in Table 6? Do they include timestamps, episode identifiers, or other location metadata? Could the authors report an oracle comparison that avoids directly revealing the target temporal location?

  1. Could the authors clarify the numerical reporting and benchmark validation protocol?

Some per-axis accuracies in Tables 7 and 9 do not appear attainable from a single set of exact-match predictions using the reported question counts. For instance, 87.1% accuracy over 97 Temporal Tracking questions is not possible after rounding a single integer correct-answer count to one decimal place. Were these results averaged across repeated inference runs, or evaluated on different subsets?

Could the authors also report independent human validation results and confidence intervals for their main comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.