acceptodds
Under review as a conference paper at ICLR 2027

VideoCogQA: A Controllable Benchmark for Cognitive Abilities in Video-Language Models

Abstract

Recent advances in Large Video-Language Models (LVLMs) have demonstrated promising progress in multimodal video understanding. Yet, it remains unclear whether these models possess the cognition-inspired capabilities required for higher-level reasoning, particularly when tasks involve symbolic manipulation and abstract reasoning. Existing benchmarks are predominantly built on annotated real-world videos, offering limited control over content composition and task difficulty and, consequently, limited diagnostic power for isolating specific cognitive abilities. To address this gap, we introduce VideoCogQA, a scalable and fully controllable benchmark inspired by game-based environments for systematically evaluating the cognitive capabilities of LVLMs. VideoCogQA leverages a programmatic video-generation engine that enables precise control over visual elements, temporal dynamics, and task complexity, thereby disentangling cognitive reasoning from reliance on prior semantic knowledge. The benchmark comprises tasks involving abstract concepts, symbolic representations, and multimodal information integration, with difficulty systematically varied through Python-based game scenarios. Extensive experiments reveal substantial limitations in current LVLMs: even state-of-the-art models such as Qwen2.5-VL-72B achieve only 48.8% average accuracy on tasks involving abstract concepts, while performance further drops by 15% as task complexity increases. These results suggest that, despite strong performance on existing video understanding benchmarks, current LVLMs still struggle with robust and systematic cognitive reasoning under controlled increases in task complexity.

Primary area:
datasets and benchmarks
Keywords:
synthetic, controllable, video-LLM, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BQqKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VideoCogQA is a controllable synthetic video question-answering benchmark with 800 videos and 3,280 questions across ten scenes and six ability categories. It uses programmatically generated environments, scene logs, and question templates to produce answers and vary task difficulty. The paper evaluates video-language models and human participants, probes performance with structured text and single-frame questions, and reports correlations and transfer experiments on natural-video benchmarks. Its main motivation is to make model failures easier to understand by reducing the ambiguity of natural-video evaluation.

Critical strengths

The motivation is compelling. A wrong answer on a video benchmark rarely tells us whether the model missed an object, forgot an event, or reasoned incorrectly. A benchmark designed around this problem could provide useful guidance beyond a single accuracy score. 2. The generator and scene logs provide a useful foundation. They allow the authors to control scene properties and derive answers from recorded states. The human study and difficulty experiments also provide some evidence that the tasks are solvable and that changing the generator settings changes their difficulty.

Critical weaknesses

The experiments do not yet provide a reliable way to distinguish the sources of failure Section 4.4 compares video input with descriptions derived from scene logs. Providing a complete log changes several things at once: it bypasses visual recognition, makes object tracking explicit, and gives the model an external record of past events. It may also expose events missed by frame sampling. The improvement therefore shows that structured information helps, but does not identify how much of the original error came from perception, memory, or reasoning. The paper acknowledges that perception is not the only possible bottleneck; however, it still lacks a diagnostic design that resolves the ambiguity motivating the benchmark. The single-frame probe in Figure 4 establishes that some attribute recognition errors exist. It does not connect those errors to the same model’s failures on the full video questions. Likewise, changing several difficulty parameters together does not establish which ability causes the decline. To address this, use the same model and questions in matched conditions: raw video; frame-aligned ground-truth object states without a compressed event history; and an explicit record of the relevant past states or events. Match the information available at the sampled timestamps, verify that intermediate states are correct, and report which individual errors each intervention repairs. Add last-frame, shuffled-frame, and question-only controls where appropriate to check whether the intended temporal information is necessary. These interventions would still need their assumptions stated; a score increase alone should not be treated as a clean causal attribution. If such validation is beyond the paper’s scope, narrow the contribution to a controllable challenge benchmark and state clearly that failure attribution remains unresolved. 2. Inconsistent results and incomplete validation weaken the empirical conclusions Several central numbers do not agree. The abstract highlights GPT-4o at 61.2% as the strongest model among the main 14 models, while Appendix Table 11 reports Gemini-3.1-Pro at 73.0%. The restricted statement is understandable, but the headline does not reflect the strongest result in the submitted paper. Figure 3 also shows GPT-4o at 66.1%, compared with 61.2% in Table 3. Update the abstract and main comparison to reflect all reported models, and identify the exact model, subset, and input settings behind each figure. More seriously, the ability correlations in Table 17 cannot be reproduced from the corresponding values in Table 16. For game-environment perception, the displayed triples [43.9, 44.3, 45.2] and [30.6, 36.2, 34.7] give Pearson r of approximately 0.537, rather than 0.995. For spatial reasoning, the displayed values give approximately 0.892, rather than 0.999. These differences directly affect the evidence for agreement with natural-video evaluation. Release the paired inputs and calculation code, regenerate the affected tables, and revise the conclusions accordingly. If different data were used, explain the mapping explicitly. Even corrected correlations across only three scales of one model family remain limited evidence, as the paper itself acknowledges. Model coverage also needs updating. Table 11 does add newer models, but important families remain absent, including Qwen3-VL (first released 23 September 2025), InternVL3.5 (26 August 2025), and GPT-5.4 (5 March 2026). All predate 26 June 2026. Evaluate representative models from these families with comparable input budgets, or give a concrete reason for their exclusion. Integrate these results with the main analysis so that conclusions about current models are based on the full evidence. The evaluation protocol needs enough detail to separate model limitations from input limitations. In particular, S10 involves sound, but the paper does not make clear what audio each model receives. Report model versions, sampled frames, resolution, audio handling, prompts, and answer extraction rules for every experiment; separate vision-only and audiovisual results where needed. For the human comparison, report model scores on the same human-tested subset and explain the difference between human playback access and model frame sampling. For the transfer experiment, specify the training split and add a matched-data-budget training baseline and repeated runs before attributing the gains to this benchmark’s particular design.

Questions for authors

The following are requests for clarification of the submitted evidence. The proposed new experiments above are revision suggestions.

  1. What exact paired observations and calculation produce Table 17? Can you reconcile them with Table 16?
  2. Does Figure 3 use the same model version, questions, and evaluation protocol as Table 3? What explains 66.1% versus 61.2%, and does the text-log comparison keep the model fixed?
  3. What audio, if any, was provided to each model for S10, and were the same questions and available information used in the human comparison?
Reviewer_jUahOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces VideoCogQA, a video question-answering benchmark built from programmatically generated, game-like scenes. The generator controls scene content and difficulty, with tasks covering perception, spatial and temporal reasoning, and symbol grounding. The paper evaluates video-language models on these tasks and analyzes how their performance changes with task difficulty and access to structured scene descriptions.

Critical strengths

The programmatic generator allows scene content and difficulty to be varied within the same task. This could be useful for examining specific failures in video understanding.

Critical weaknesses
  1. The paper is hard to follow. Figure 2 is an example: the small panels have coordinate grids, but their x- and y-axes are not explained. I had to work out how to read the figure myself. Section 3.2 then says "We describe the ten scenes below" and gives a full page of ten numbered scene descriptions. I do not think this is a good use of the main text. The detailed descriptions could go in the appendix, leaving room to explain the benchmark design through a few representative examples. I suggest looking at how last year's ICLR best paper organizes its exposition.

  2. The baseline coverage is insufficient, particularly for recent video-understanding models. The abstract highlights GPT-4o as "the strongest model," and the Qwen models in Table 3 stop at Qwen2.5-VL. The evaluation feels dated. I would like to see a broader comparison that includes Qwen3-VL and other newer video-understanding models. For a benchmark paper, I need to know what current models still struggle with; the present model set does not give me enough evidence to judge that.

Questions for authors
  1. Do the authors have results for Qwen3-VL or other newer video-understanding models? Do the reported failure patterns and difficulty trends still hold for these models?
  2. What do the x- and y-axes within each panel of Figure 2 represent? Please clarify how the panels should be read.
Reviewer_DpdbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a controllable benchmark for evaluating large video-language models (LVLMs) that can systematically assess temporal, abstract, and cognitive abilities in video understanding across varying difficulty levels. They argue that existing benchmarks mostly rely on natural videos, which entangle multiple dimensions together, making it challenging to identify which specific capabilities models are failing on, e.g., object perception, action perception, temporal reasoning, etc. To address this, they construct VideoCogQA, a synthetic video benchmark developed using a programmatic game-based generator. Since the videos are generated synthetically, explicit controls over object attributes, temporal dynamics, scene structure, and difficulty are used to construct the QA pairs. In total, it contains 800 videos constituting approximately 3.3K MCQs.

They evaluate a diverse set of LVLMs (including both open-source and closed-source models) on their benchmark, and find that on average, model performance lags behind human-level performance. An interesting finding is that model performance decreases proportionally with increasing difficulty level (e.g., easy vs. hard splits), which implicitly validates the controllability of the benchmark splits. Additional analysis is conducted to study the potential causes of limited performance. Specifically, the finding that model performance improves substantially when provided with structured visual extraction points suggests that extracting visual semantics from frames is one key bottleneck. Other experiments, such as transfer experiments and comparisons with existing benchmarks, are also informative.

Critical strengths

From a literature standpoint, this work highlights an important gap in existing LVLM benchmarks, namely the lack of task and difficulty isolation in natural videos. Using such a benchmark, researchers developing LVLMs can explicitly examine model capabilities without introducing confounders that would blur their evaluations. This work is therefore clearly positioned within the current landscape of video benchmarking literature.

Another strength of this benchmark is that it is easily scalable while keeping its evaluation dimensions, such as difficulty and motion types, explicitly controllable. Based on my knowledge of the literature, this emphasis is relatively uncommon in existing benchmarks.

Lastly, the paper is clearly presented and easy to follow. Additional experiments such as transfer results and frame-level ablations are informative.

Critical weaknesses

One concern I have is that the evaluation videos are purely synthetic, while the pretraining data of most LVLMs constitutes real-world videos. Therefore, I wonder whether these evaluation videos are simply out-of-distribution (OOD) for most LVLMs, which could also be a contributing reason for their suboptimal performance. This concern is slightly mitigated for closed-source models, where I can imagine the pretraining data to cover multiple distributions including synthetic ones. An analysis or discussion on this would help quite a bit.

Secondly, in the main results section (Experiment 4.3), the breakdown of results is mainly performed on the averaged model performance level which is not very informative. As different LVLMs use different architectures, datasets and pretraining schedules, a dimension-wise analysis across models is somewhat missing. What are model family specific (e.g., open-source vs closed source, initially released LVLMs vs recent ones etc) trends with respect to different dimensions of the benchmark? A discussion on this is highly recommended.

Lastly, I did not see qualitative examples of model predictions across different samples from different dimensions of the benchmark. Qualitative results provide complementary findings alongside empirical results.

There are some writing-related issues that did not factor into my rating or decision, which I am sharing below:

There are quite a few sentences in the paper that are very general or sometimes unclear, and not sufficiently grounded or concrete.

For example, in line 213, the phrase “latent scene state” is a bit ambiguous. What does it mean exactly? Latents usually refer to the hidden states of a neural network. Is that what is being referred to here? In lines 39-40, “source of an error is” is unclear. While I understand it refers to a mistake made by the LVLM, it does not come across clearly and can be confusing.

Questions for authors

Did the authors observe any failure cases of their benchmark (e.g., shortcomings in the pipeline for generating videos and corresponding QA templates)? A discussion on future work would help improve the manuscript and benefit researchers who would like to work in this direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.