- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
- Motivation: Existing video benchmarks mainly focus on static perception or isolated tasks, while higher-level temporal cognitive abilities such as object permanence, episodic memory, causal reasoning, and planning remain under-evaluated.
- Benchmark: The paper introduces CogViD, which organizes video understanding into 6 cognitive levels (L0–L5), from static perception to planning and metacognition. It contains 25,505 QA pairs, 2,461 videos, 159.7 hours of footage, and 58 sub-tasks.
- Data Construction: CogViD uses a multi-expert annotation pipeline: different models perform video description, template-based QA generation, and independent verification, followed by filtering and human review to improve data quality.
- Key Findings: Current MLLMs perform relatively well on basic recognition but struggle with counting, maintaining hidden states, object permanence, episodic memory, and high-level goal/error reasoning. Long-video models also show no consistent memory advantage, suggesting that processing more frames does not necessarily lead to stronger temporal cognition.
Critical strengths
- Comprehensive cognitive framework: Organizes video understanding into 6 cognitive levels and 58 sub-tasks, providing more systematic coverage than flat task-based benchmarks.
- Large and diverse benchmark: Includes 25,505 QA pairs over 2,461 real-world videos, covering both multiple-choice and open-ended questions.
- Careful annotation pipeline: Uses separate models for QA generation and verification, followed by filtering and human review, reducing self-validation bias.
Critical weaknesses
- Heavy reliance on LLM-generated annotations: Despite verification and human review, the benchmark still depends substantially on model-generated video descriptions and QA pairs.
- Cross-level comparison is not fully controlled: Different cognitive levels are generally evaluated on different videos, so performance differences may partly reflect differences in video difficulty rather than cognitive ability alone.
- The cognitive hierarchy is not strictly monotonic in difficulty: Higher levels are not always harder in practice; for example, some L2/L4 tasks outperform L1/L3, making the interpretation of the six levels less straightforward
Questions for authors
-
Annotation Quality: Since the benchmark relies heavily on LLM-generated video descriptions and QA pairs, how do the authors ensure annotation accuracy, particularly for complex cognitive tasks such as causal reasoning and episodic memory? Could the authors provide more details on the human verification process and annotation agreement?
-
Cross-Level Comparison: Different cognitive levels are evaluated using different videos and questions, which may introduce differences in difficulty. How do the authors disentangle cognitive ability from variations in video complexity and question difficulty? Have any controlled comparisons been conducted across cognitive levels using the same videos?
-
Cognitive Hierarchy: The six cognitive levels are designed to represent progressively higher-level cognitive abilities, but the experimental results do not always show increasing difficulty across levels. How do the authors interpret this observation? Does it suggest that the proposed hierarchy reflects different types of cognitive abilities rather than a strictly increasing difficulty structure?