acceptodds
Under review as a conference paper at ICLR 2027

CogViD: A Comprehensive Cognitive Benchmark for Evaluating Video MLLMs

12 views

Abstract

Video understanding benchmarks have evolved from short clips to hour-long videos and from single to dozens of tasks, pushing multimodal large language models (MLLMs) toward long-form, multi-task comprehension. While existing benchmarks provide reasonable coverage of static perception, object permanence and episodic-like memory receive only partial coverage and often lack high-level planning via dedicated sub-tasks. In this paper, we propose CogViD, a cognition-inspired dataset that introduces new tasks and provides a comprehensive benchmark organized into hierarchical levels. CogViD contains QA pairs over videos and hours of footage, spanning sub-tasks and questions in both multiple-choice and open-ended form. Furthermore, we propose a template-driven multi-expert annotation pipeline in which level-specific templates drive QA generation, and verification is assigned to separate models to mitigate the bias of a model validating its own outputs, and blind-answerable and near-duplicate answers are filtered before human review. Our comprehensive evaluation across state-of-the-art MLLMs reveals a large and consistent gap in maintaining quantities and states over time (including counting over different frames), procedural memory to identify missing steps, and open-ended goal and error reasoning. CogViD not only offers a robust benchmark but also paves the way for future research toward human-like intelligence in video understanding.

Primary area:
datasets and benchmarks
Keywords:
benchmark, video understanding, datasets, evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hEhZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • Motivation: Existing video benchmarks mainly focus on static perception or isolated tasks, while higher-level temporal cognitive abilities such as object permanence, episodic memory, causal reasoning, and planning remain under-evaluated.
  • Benchmark: The paper introduces CogViD, which organizes video understanding into 6 cognitive levels (L0–L5), from static perception to planning and metacognition. It contains 25,505 QA pairs, 2,461 videos, 159.7 hours of footage, and 58 sub-tasks.
  • Data Construction: CogViD uses a multi-expert annotation pipeline: different models perform video description, template-based QA generation, and independent verification, followed by filtering and human review to improve data quality.
  • Key Findings: Current MLLMs perform relatively well on basic recognition but struggle with counting, maintaining hidden states, object permanence, episodic memory, and high-level goal/error reasoning. Long-video models also show no consistent memory advantage, suggesting that processing more frames does not necessarily lead to stronger temporal cognition.
Critical strengths
  1. Comprehensive cognitive framework: Organizes video understanding into 6 cognitive levels and 58 sub-tasks, providing more systematic coverage than flat task-based benchmarks.
  2. Large and diverse benchmark: Includes 25,505 QA pairs over 2,461 real-world videos, covering both multiple-choice and open-ended questions.
  3. Careful annotation pipeline: Uses separate models for QA generation and verification, followed by filtering and human review, reducing self-validation bias.
Critical weaknesses
  1. Heavy reliance on LLM-generated annotations: Despite verification and human review, the benchmark still depends substantially on model-generated video descriptions and QA pairs.
  2. Cross-level comparison is not fully controlled: Different cognitive levels are generally evaluated on different videos, so performance differences may partly reflect differences in video difficulty rather than cognitive ability alone.
  3. The cognitive hierarchy is not strictly monotonic in difficulty: Higher levels are not always harder in practice; for example, some L2/L4 tasks outperform L1/L3, making the interpretation of the six levels less straightforward
Questions for authors
  1. Annotation Quality: Since the benchmark relies heavily on LLM-generated video descriptions and QA pairs, how do the authors ensure annotation accuracy, particularly for complex cognitive tasks such as causal reasoning and episodic memory? Could the authors provide more details on the human verification process and annotation agreement?

  2. Cross-Level Comparison: Different cognitive levels are evaluated using different videos and questions, which may introduce differences in difficulty. How do the authors disentangle cognitive ability from variations in video complexity and question difficulty? Have any controlled comparisons been conducted across cognitive levels using the same videos?

  3. Cognitive Hierarchy: The six cognitive levels are designed to represent progressively higher-level cognitive abilities, but the experimental results do not always show increasing difficulty across levels. How do the authors interpret this observation? Does it suggest that the proposed hierarchy reflects different types of cognitive abilities rather than a strictly increasing difficulty structure?

Reviewer_rdLZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed a video understanding benchmark called CogVid, comprising 25k QA pairs over 2.4k videos and 58 subtasks. The benchmark organizes these tasks into six cognitive-inspired levels, ranging from static perception to planning and metacognition. To improve the overall quality and quantity, the authors combined template-driven question generation, verification by separate models, and human review. The evaluation of large number of open-sourced MLLMs and proprietary models identifies recurring weaknesses in quantitative object permanence, event counting, and open-ended goal and error reasoning. The benchmark offers broad task coverage and useful diagnostic observations. However, the central claim that its organization captures a meaningful hierarchy of cognitive abilities is not sufficiently substantiated, and some interpretations of the experimental results require more controlled evidence.

Critical strengths
  1. As shown in Table 1, CogViD covers more subtasks than the benchmarks, and it includes both multiple-choice and open-ended questions. Its coverage of object permanence, procedural structure, and episodic-like memory provides a useful complement to conventional video understanding evaluations.
  2. The breakdowns distinguish qualitative from quantitative object permanence and identify differences between event counting and other forms of episodic recall. Comparing recurrence counting with frame counting is also a useful step toward understanding whether counting failures are specific to temporally extended inputs.
  3. The scalable construction pipeline with multiple quality-control stages is good. The use of task-specific templates, separate models for generation and verification, and human review provides a structured approach to dataset construction.
Critical weaknesses
  1. The contribution of the cognitive framework beyond task categorization remains unclear. The paper motivates CogViD through cognitive theories and dependencies between abilities (L78–81 and L188–193), but does not sufficiently explain how these foundations establish the particular six-level ordering. For example, why should procedural memory precede episodic-like memory, and what testable dependency does this ordering imply? Grouping 58 subtasks into six categories provides an organizational taxonomy, but does not by itself establish a progressive cognitive hierarchy.

  2. Related to Q1, the shared performance patterns discussed in L399–401 demonstrate common strengths and weaknesses across models, but do not establish the proposed relationships between levels. Moreover, most cross-level comparisons use different videos, as acknowledged in L361–363, leaving content and evidence difficulty as possible confounds. One minor concern that does not affect my scoring is that L5 is evaluated through open-ended generation, whereas L0–L4 use multiple-choice scoring (L207–208), possibly complicating the interpretation of its lower scores.

  3. The counting analysis does not sufficiently isolate the proposed failure mechanism. Figure 6(a,c) shows systematic underestimation, but this alone does not establish that models rely on a small-count prior or that maintaining a count, rather than temporal forgetting, is the principal bottleneck (L457–461). A useful diagnostic would evaluate downstream questions that require counting as an intermediate step, comparing the original input, model-predicted intermediate counts, and ground-truth intermediate counts. This would help distinguish failures in obtaining quantities from failures in using them. Such an intervention should be applied to questions whose final answers are not the counts themselves.

  4. Table 2 includes Qwen3-VL, MiniCPM-V-4.5, and a model listed as GPT-Astra, so the evaluation is not restricted to older models. Nevertheless, much of the model coverage—and particularly the four-model counting probe—relies on older ones. Including representative newer open models, such as InternVL3 and Qwen3.5 (but not limited to them), would help establish whether the main diagnostic findings remain applicable. Input budgets also vary substantially: Qwen3-VL uses 8 frames, while Qwen2.5-VL uses 256. Controlled frame-budget comparisons or sensitivity analyses would make conclusions about temporal memory and counting more convincing.

  5. Minor presentation concern: Figure 6 groups different analyses awkwardly. Panels (a) and (c) directly compare recurrence and frame counting, whereas panel (b) presents overall performance across cognitive levels. Moving panel (b) to the level-wise results discussion would make the counting comparison easier to follow.

Questions for authors
  1. What precisely does the proposed cognitive hierarchy represent? Does the L0–L5 ordering imply prerequisite relationships between abilities, increasing temporal evidence requirements, or simply a categorization of tasks?
  2. To what extent do counting errors explain failures in downstream reasoning? Please refer to Q4 for details.
Reviewer_Gb67OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces CogViD, a video understanding benchmark containing 25,505 QA pairs across 2,461 videos and 159.7 hours of footage. The benchmark organizes 58 subtasks into six proposed cognitive levels, spanning static perception (L0), object permanence (L1), procedural memory (L2), episodic-like memory (L3), causal reasoning (L4), and planning and metacognition (L5).

The authors also introduce a multi-model annotation pipeline using Gemini 3 Pro for video descriptions, Claude Opus for question generation, and GPT-4o and GPT-5.2 for verification, followed by human review.

The paper evaluates 25 video MLLMs and reports substantial performance differences across cognitive levels and subtasks. Its principal empirical findings concern models' difficulty with counting, maintaining object states through occlusion, recalling event occurrences, and answering open-ended planning questions.

Critical strengths
  1. Broad coverage of video-understanding capabilities.

The benchmark covers an unusually broad collection of capabilities, with 58 subtasks organized around a common taxonomy. I appreciate the emphasis on object permanence, state tracking, procedural understanding, and episodic memory, which are important for video understanding but are not always evaluated in sufficient detail.

The dataset is relatively large, and its combination of multiple-choice and open-ended questions could make it a useful evaluation resource.

  1. Useful fine-grained analyses of model failures.

I found the analyses of counting particularly interesting. The authors show that models struggle both with recurrence counting across time and counting objects within frames, suggesting that the limitation may not be attributable exclusively to long-range memory.

The breakdown of object permanence into qualitative and quantitative tasks is also useful. Such analyses provide more actionable information than aggregate video QA accuracy alone.

Critical weaknesses
  1. The proposed cognitive hierarchy is not convincingly established by the experimental design.

The main conceptual contribution of the paper is its six-level, cognitive theory-inspired hierarchy. However, the experiments do not convincingly establish that these levels constitute a progressively more complex set of capabilities, or that performance differences can be attributed to the cognitive demands assigned to each level.

There are three related concerns.

First, the different levels are evaluated on almost entirely different videos. The paper reports 2,468 level assignments for 2,461 videos, meaning that there is very little overlap across levels. This creates a substantial confound: the videos are collected using level-specific keywords and differ in content, duration, and domain. Consequently, differences in accuracy may reflect differences in video difficulty rather than cognitive complexity.

Second, L0–L4 are evaluated using multiple-choice questions, whereas L5 is evaluated exclusively using open-ended generation and an LLM-based judge. The results are therefore not directly comparable. The paper repeatedly interprets the low accuracy on L5 as evidence that planning and metacognition represent particularly difficult cognitive capabilities. However, the observed gap could arise partly from response format, generation behavior, evaluation criteria, or judge bias. Appendix C even identifies some very low L5 scores as generation failures rather than failures of the underlying cognitive capability.

Third, the proposed hierarchy does not consistently produce a monotonic progression in difficulty. For many models, L2 and L4 achieve higher scores than L1 and L3. This does not invalidate the value of the individual categories, but it calls into question the interpretation of the six levels as a progression in cognitive complexity.

A related concern is that the benchmark's own evidence requirements appear inconsistent. Appendix B explicitly requires that every question use multiple distinct moments and excludes questions answerable from a single frame. However, the L0 taxonomy includes tasks such as scene classification, object recognition, OCR, and counting in frame, which are explicitly defined around individual frames. The L0 example in Figure 1 also appears answerable from a single static observation.

Overall, the paper establishes a useful taxonomy of capabilities, but not a validated hierarchy of cognitive abilities. This distinction matters because the hierarchy is presented as the paper's main conceptual contribution.

  1. The benchmark validation and evaluation methodology leave important questions unanswered.

The annotation pipeline combines several capable models with additional filtering and human review. However, the paper does not provide sufficient evidence to establish the quality of the resulting benchmark, particularly for the more subjective reasoning and planning tasks.

The reported 94.8% description fidelity is based on 500 audited video segments and measures the accuracy of intermediate summaries rather than the validity of the final QA pairs. More importantly, the extent of human validation is unclear. Section 3.5 describes human review of the generated questions, while the AI Use Statement mentions manual auditing of only a random subset of 100 questions. I could not find detailed question-level acceptance rates, inter-annotator agreement, or error rates after human verification.

This is particularly consequential for the L4 and L5 tasks, where causal explanations, hypothetical outcomes, goals, and alternative plans may admit multiple reasonable interpretations.

The evaluation methodology also introduces additional difficulties:

The dataset contains 11,773 open-ended questions, but 8,313 of these belong to L1–L4 and are not reflected in the main reported evaluation. Instead, only MCQs are scored for L0–L4 and only open-ended questions for L5. It is unclear why the available open-ended questions at lower levels are excluded.

The paper reports L5 performance using GPT-4-based evaluation but provides insufficient detail about the grading rubric, calibration against human judgments, and judge reliability.

The overall score is effectively a macro-average across six levels, while the per-level scores are unweighted averages across subtasks. Because the number of questions differs substantially between levels and subtasks, the resulting aggregate is not straightforwardly interpretable as overall question accuracy.

Models receive substantially different numbers of input frames, ranging from 8 to 256. Consequently, observations about model scaling, long-video architectures, and memory capacity are difficult to disentangle from differences in visual evidence availability.

Finally, the novelty positioning needs greater care. Video-MME-v2 already proposes a progressive multilevel evaluation hierarchy, and benchmarks such as VideoVista and VideoReasonBench cover several temporal reasoning and tracking capabilities. In particular, Table 1 appears to understate existing coverage by assigning VideoVista exclusively to L0, despite its explicit inclusion of multiple reasoning tasks.

I view the extensive taxonomy as a useful organizational contribution, but the evidence for its novelty as a cognitive hierarchy, and for the reliability of its evaluation, is insufficient in the current submission.

Questions for authors
  1. How can the authors establish that the six cognitive levels form a hierarchy rather than a taxonomy of different task types?

Would it be possible to evaluate multiple levels on the same videos, or to construct a controlled subset where video content and answer format are held fixed? Such an analysis would help distinguish cognitive complexity from content difficulty.

  1. Can the authors provide comparable evaluations across question formats?

Since L1–L4 already contain open-ended questions, could the authors evaluate them using the same open-ended protocol applied to L5? Conversely, a multiple-choice version of representative L5 questions would help determine how much of the performance gap is attributable to format rather than cognitive difficulty.

  1. What is the actual quality-control protocol for the final dataset?

How many of the 25,505 questions were manually verified against video evidence? What are the acceptance and correction rates? Were multiple annotators used, and was their agreement measured? How was the LLM judge validated on open-ended tasks with potentially multiple acceptable answers?

  1. How are the multi-moment evidence constraints enforced?

Appendix B requires every question to depend on multiple moments, yet several L0 task definitions appear explicitly single-frame. What proportion of the dataset can be answered from one frame? Has this requirement been independently tested rather than only enforced through generation prompts?

  1. Could the authors clarify the AI Use Statement?

The statement says that dataset construction and evaluation were conducted without generative AI tools, whereas the main paper describes using Gemini, Claude, GPT-4o, and GPT-5.2 for precisely these purposes. Is this a wording error?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.