- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HAVEN, a long video benchmark that provides linked frame, shot, and video-level annotations for evaluating summarization, temporal ordering, saliency, and grounding on a shared collection of 365 videos with human-audited textual annotations. Reported results show task-dependent model performance. Strong summary scores do not consistently imply accurate visual selection or temporal reasoning.
Critical strengths
- Useful annotation. Explicit cross-granularity links could support more interpretable evaluation than answer accuracy alone.
- Good evaluation and analysis of the video understanding capabilities of VLMs.
- Released resource. The reported human auditing and released data and code are valuable.
Critical weaknesses
- The gap relative to existing benchmarks is overstated. Although using QA formats, LVBench, Video-MME, and LongVideoBench already cover local details, summary, broader temporal reasoning and more. A shared QA format does not imply single-granularity understanding. This paper needs clearer evidence of additional diagnostic value.
- The novelty requires re-positioning. The earlier HAVEN framework (https://arxiv.org/pdf/2601.13719 in CVPR 2026) already proposes hierarchical representations and cross-modal associations. The benchmark contribution is distinct, but should be justified through its annotations and evaluation protocol, not hierarchy alone.
- Reference reliability remains uncertain. The generated annotations are not ground-truth. Missing facts, incorrect descriptions, or valid alternative summaries can distort reference-based scores. Human auditing helps, but the independent study covers only 20 videos, with summary relevance averaging only 3.19/5.
- Several comparisons are confounded. Differences in frame coverage, summary-length requirements, and frame-budget enforcement complicate interpretation. Matched protocols, simple non-semantic baselines, and human performance estimates would strengthen the conclusions.
Questions for authors
- What failure modes does HAVEN reveal that are not adequately captured by LVBench, Video-MME, and LongVideoBench? Please provide concrete examples or a capability comparison distinguishing annotation granularity from reasoning granularity.
- Which annotations, alignment targets, or evaluation procedures constitute the main advancement over the earlier HAVEN (CVPR'26) framework? Please clarify the contribution beyond hierarchical organization and cross-modal associations.
- How are factual errors, missing information, and valid alternative summaries or grounding targets handled?
- Which conclusions remain when frame coverage, summary length, and selection budgets are matched? Can you provide simple random or position-based baselines, and human performance estimates where available, to support the reported scores?