acceptodds
Under review as a conference paper at ICLR 2027

HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding

Abstract

Recent research often evaluates models’ video understanding abilities through tasks such as summarization, temporal reasoning, and cross-modal reasoning. Existing benchmarks typically focus on a single task and a single level of granularity for each video, limiting their ability to provide a comprehensive evaluation of video understanding. We introduce HAVEN, a benchmark that links visual and textual annotations at the frame, shot, and video levels. Built from six existing video-summarization datasets, HAVEN contains 11,055 videos and uses a 365-video subset with human-audited textual annotations for standard evaluation. Its task suite covers summarization, temporal ordering, cross-modal grounding, and saliency ranking. Empirical experiment results reveal a comprehensive and fine-grained evaluation of the video understanding capabilities of widely used vision-language models. These observations show that textual-summary scores alone do not capture the variation measured by visual selection and alignment tasks. HAVEN provides a shared annotation structure for examining these complementary aspects of video understanding.

Primary area:
datasets and benchmarks
Keywords:
Video Understanding, Benchmark, MLLMs

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_AtArOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HAVEN, a long video benchmark that provides linked frame, shot, and video-level annotations for evaluating summarization, temporal ordering, saliency, and grounding on a shared collection of 365 videos with human-audited textual annotations. Reported results show task-dependent model performance. Strong summary scores do not consistently imply accurate visual selection or temporal reasoning.

Critical strengths
  1. Useful annotation. Explicit cross-granularity links could support more interpretable evaluation than answer accuracy alone.
  2. Good evaluation and analysis of the video understanding capabilities of VLMs.
  3. Released resource. The reported human auditing and released data and code are valuable.
Critical weaknesses
  1. The gap relative to existing benchmarks is overstated. Although using QA formats, LVBench, Video-MME, and LongVideoBench already cover local details, summary, broader temporal reasoning and more. A shared QA format does not imply single-granularity understanding. This paper needs clearer evidence of additional diagnostic value.
  2. The novelty requires re-positioning. The earlier HAVEN framework (https://arxiv.org/pdf/2601.13719 in CVPR 2026) already proposes hierarchical representations and cross-modal associations. The benchmark contribution is distinct, but should be justified through its annotations and evaluation protocol, not hierarchy alone.
  3. Reference reliability remains uncertain. The generated annotations are not ground-truth. Missing facts, incorrect descriptions, or valid alternative summaries can distort reference-based scores. Human auditing helps, but the independent study covers only 20 videos, with summary relevance averaging only 3.19/5.
  4. Several comparisons are confounded. Differences in frame coverage, summary-length requirements, and frame-budget enforcement complicate interpretation. Matched protocols, simple non-semantic baselines, and human performance estimates would strengthen the conclusions.
Questions for authors
  1. What failure modes does HAVEN reveal that are not adequately captured by LVBench, Video-MME, and LongVideoBench? Please provide concrete examples or a capability comparison distinguishing annotation granularity from reasoning granularity.
  2. Which annotations, alignment targets, or evaluation procedures constitute the main advancement over the earlier HAVEN (CVPR'26) framework? Please clarify the contribution beyond hierarchical organization and cross-modal associations.
  3. How are factual errors, missing information, and valid alternative summaries or grounding targets handled?
  4. Which conclusions remain when frame coverage, summary length, and selection budgets are matched? Can you provide simple random or position-based baselines, and human performance estimates where available, to support the reported scores?
Reviewer_sydfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HAVEN, a video-understanding benchmark built from six existing video-summarization datasets (11,055 videos). Its visual and textual annotations are organized into a frame–shot–video hierarchy, with explicit links between summary sentences and their source shots. Importance labels come from the source datasets. Textual annotations are generated by models (Video-LLaVA-7B writes the frame descriptions; LLMs/MLLMs write the shot and video-level text) and are human-audited for a 365-video evaluation subset. The benchmark defines six tasks: text summarization, keyframe selection, joint video-text summarization, temporal ordering, sentence-to-shot grounding, and saliency ranking. The authors evaluate several 7–8B open-source video MLLMs, GPT-5.2, GLM-4.6V-Flash, and a text-only proxy. They find that no model leads across tasks and that textual-summary scores do not predict performance on visual selection, temporal, or grounding tasks.

Critical strengths
  1. Unified hierarchical annotation. HAVEN links frame-, shot-, and video-level annotations, including explicit sentence-to-shot links, so several complementary tasks can be evaluated on the same videos. This shared structure is the paper’s main contribution, and it supports the finding that textual-summary scores do not reflect visual grounding ability.

  2. Intuitive and useful tasks. Temporal ordering and sentence-to-shot grounding directly test whether models ground their language in visual evidence, rather than just producing text that resembles a reference.

Critical weaknesses
  1. The quality and consistency of the annotations and references is not well established.

The frame descriptions come from Video-LLaVA-7B, a relatively weak and outdated model, and the paper does not justify this choice. The references may inherit that model's capability ceiling and phrasing style. Importance labels from six sources with different definitions and granularities are mapped onto the same 15% budget and metrics. The paper doesn't explain this conversion enough, and it reports no per-dataset results. The independent quality check covers only 20 videos. Summary relevance (3.19/5) and conciseness (3.43/5) are low, and inter-rater agreement is weak, so it does not support the overall quality claims.

  1. The evaluation does not yet show that the benchmark can tell models apart.

Few models are evaluated, mostly at 7–8B scale, with no larger or newer models such as larger Qwen3-VL variants. It remains unclear whether HAVEN separates model generations and scales. V2T relies on BERTScore/ROUGE similarity to references produced by a particular generation pipeline. Accurate summaries written at a different granularity may be unfairly penalized. There is no ceiling or discriminability analysis for temporal ordering and grounding, so it is unclear whether these tasks give an informative signal for current models.

Questions for authors

Why was Video-LLaVA-7B chosen to generate the frame descriptions? Which MLLM and LLM generate the shot descriptions, documents, and summaries? What fraction of the generated text was corrected or regenerated during the human audit?

How are the different source labels (binary selections, continuous scores, and viewer-interaction signals) converted to the shared 15% budget? Could you report results separately for each source dataset?

Summary relevance and conciseness receive low ratings, and inter-rater agreement is weak. How do you justify using these summaries as reference answers?

Reviewer_m62gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HAVEN, a benchmark that rebuilds six existing video summarization datasets into a shared frame, shot, and video hierarchy. Each level gets LLM-generated text descriptions, and every sentence in the video-level document and summary is linked to the shots it was built from. A 365-video subset is human-audited and used to evaluate MLLMs on summarization, temporal ordering, sentence-to-shot grounding, and saliency ranking. The authors argue that textual summary scores do not reflect a model’s ability to select or ground visual evidence, and that models show different strengths across tasks.

Critical strengths

S1. Sentence-to-shot links are recorded at construction time. Summary sentences are assembled from shot descriptions and keep their source shot IDs, so grounding can be evaluated without any post-hoc alignment. The same videos also support summarization, ordering, grounding, and ranking. This is the paper’s main practical contribution and the main reason I see value in the resource.

S2. The quality of the reference annotations is reported openly. The evaluation subset was audited twice, and a separate group of raters scored the final annotations, with agreement statistics and an LLM-judge comparison. The authors also show that the summary level is the least reliable. For a benchmark whose references start from LLM outputs, this is what lets a reader judge how far to trust them.

Critical weaknesses

W1. The evaluation data is not well documented. Table 1 lists 5,590 Mr.HiSum and 5,300 VideoXum videos, but the original datasets have 31,892 and 14,001, and neither number matches an official split. YouTube has 40 videos instead of the usual 39. The paper gives neither the filtering criteria nor the source split of the evaluation videos. Without them, readers cannot check how representative the benchmark is or whether it overlaps with model training data (VideoXum comes from ActivityNet).

W2. The main claim is not well supported. Random V2V F1 is about 0.15, while the text proxy and Qwen3-VL reach only 0.171 and 0.173. Ordering and ranking ρ values are mostly below 0.3 on just 5 to 8 shots, and failed outputs are dropped without reported failure rates. The text proxy’s 0.804 grounding F1 largely reflects text-to-text matching, and the V2VT changes appear driven by how many frames each model outputs (Table 4). It is unclear whether the reported model differences are real.

W3. The paper does not position itself against close prior work. The V2T, V2V, and V2VT split and VT-CLIPScore come from VideoXum (Lin et al., 2024), which is cited only as a data source. V2Xum-LLM (Hua et al., 2025) is not cited. Shot2Story (Han et al., 2025) and Ego4D-HCap (Islam et al., 2024) already pair shot- or segment-level captions with video summaries, but the paper does not compare against them. This makes the novelty of the hierarchy and alignment hard to assess.

Questions for authors

Q1. What criteria were used to keep 5,590 Mr.HiSum and 5,300 VideoXum videos, and which original split did the 100 evaluation videos from each come from? Why does YouTube have 40 videos?

Q2. What are the per-model parsing failure rates for temporal ordering, saliency ranking, and grounding? Does the ranking in Table 3 hold if failures are scored as 0?

Q3. Can you add random-selection baselines for V2V F1 and ρ, and confidence intervals for the main numbers (for example, video-level bootstrap)?

Q4. Beyond the source data, what does HAVEN offer that Shot2Story and Ego4D-HCap do not? Why are the V2T, V2V, and V2VT definitions not attributed to VideoXum?

Q5. How many frames does each model actually output on average in V2VT?

Reviewer_MwnfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary
  1. The authors extend six existing video-summarization datasets with visual and textual annotations at three levels : frame, shot, and video levels to evaluate how well multimodal LLMs understand videos.
  2. The authors find that multimodal LLMs differ in zero-shot evaluations across their capabilities in video summarization, temporal ordering, multimodal grounding, and saliency ranking.
Critical strengths

The benchmark allows complementary evaluation of video-understanding capabilities (video summarization, temporal ordering, multimodal grounding, and saliency ranking) on the same videos.

Critical weaknesses
  1. The paper essentially combines existing tasks on the same set of videos, but it does not clearly demonstrate what additional insight comes from its hierarchical annotation structure or evaluation.
  2. The choice of models for evaluation lack a clear rationale, and the absence of systematic failure analysis leaves it unclear why different MLLMs fail on different tasks or how the findings could guide future model training and data curation.
Questions for authors

see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.