- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
VideoCogQA is a controllable synthetic video question-answering benchmark with 800 videos and 3,280 questions across ten scenes and six ability categories. It uses programmatically generated environments, scene logs, and question templates to produce answers and vary task difficulty. The paper evaluates video-language models and human participants, probes performance with structured text and single-frame questions, and reports correlations and transfer experiments on natural-video benchmarks. Its main motivation is to make model failures easier to understand by reducing the ambiguity of natural-video evaluation.
Critical strengths
The motivation is compelling. A wrong answer on a video benchmark rarely tells us whether the model missed an object, forgot an event, or reasoned incorrectly. A benchmark designed around this problem could provide useful guidance beyond a single accuracy score. 2. The generator and scene logs provide a useful foundation. They allow the authors to control scene properties and derive answers from recorded states. The human study and difficulty experiments also provide some evidence that the tasks are solvable and that changing the generator settings changes their difficulty.
Critical weaknesses
The experiments do not yet provide a reliable way to distinguish the sources of failure Section 4.4 compares video input with descriptions derived from scene logs. Providing a complete log changes several things at once: it bypasses visual recognition, makes object tracking explicit, and gives the model an external record of past events. It may also expose events missed by frame sampling. The improvement therefore shows that structured information helps, but does not identify how much of the original error came from perception, memory, or reasoning. The paper acknowledges that perception is not the only possible bottleneck; however, it still lacks a diagnostic design that resolves the ambiguity motivating the benchmark. The single-frame probe in Figure 4 establishes that some attribute recognition errors exist. It does not connect those errors to the same model’s failures on the full video questions. Likewise, changing several difficulty parameters together does not establish which ability causes the decline. To address this, use the same model and questions in matched conditions: raw video; frame-aligned ground-truth object states without a compressed event history; and an explicit record of the relevant past states or events. Match the information available at the sampled timestamps, verify that intermediate states are correct, and report which individual errors each intervention repairs. Add last-frame, shuffled-frame, and question-only controls where appropriate to check whether the intended temporal information is necessary. These interventions would still need their assumptions stated; a score increase alone should not be treated as a clean causal attribution. If such validation is beyond the paper’s scope, narrow the contribution to a controllable challenge benchmark and state clearly that failure attribution remains unresolved. 2. Inconsistent results and incomplete validation weaken the empirical conclusions Several central numbers do not agree. The abstract highlights GPT-4o at 61.2% as the strongest model among the main 14 models, while Appendix Table 11 reports Gemini-3.1-Pro at 73.0%. The restricted statement is understandable, but the headline does not reflect the strongest result in the submitted paper. Figure 3 also shows GPT-4o at 66.1%, compared with 61.2% in Table 3. Update the abstract and main comparison to reflect all reported models, and identify the exact model, subset, and input settings behind each figure. More seriously, the ability correlations in Table 17 cannot be reproduced from the corresponding values in Table 16. For game-environment perception, the displayed triples [43.9, 44.3, 45.2] and [30.6, 36.2, 34.7] give Pearson r of approximately 0.537, rather than 0.995. For spatial reasoning, the displayed values give approximately 0.892, rather than 0.999. These differences directly affect the evidence for agreement with natural-video evaluation. Release the paired inputs and calculation code, regenerate the affected tables, and revise the conclusions accordingly. If different data were used, explain the mapping explicitly. Even corrected correlations across only three scales of one model family remain limited evidence, as the paper itself acknowledges. Model coverage also needs updating. Table 11 does add newer models, but important families remain absent, including Qwen3-VL (first released 23 September 2025), InternVL3.5 (26 August 2025), and GPT-5.4 (5 March 2026). All predate 26 June 2026. Evaluate representative models from these families with comparable input budgets, or give a concrete reason for their exclusion. Integrate these results with the main analysis so that conclusions about current models are based on the full evidence. The evaluation protocol needs enough detail to separate model limitations from input limitations. In particular, S10 involves sound, but the paper does not make clear what audio each model receives. Report model versions, sampled frames, resolution, audio handling, prompts, and answer extraction rules for every experiment; separate vision-only and audiovisual results where needed. For the human comparison, report model scores on the same human-tested subset and explain the difference between human playback access and model frame sampling. For the transfer experiment, specify the training split and add a matched-data-budget training baseline and repeated runs before attributing the gains to this benchmark’s particular design.
Questions for authors
The following are requests for clarification of the submitted evidence. The proposed new experiments above are revision suggestions.
- What exact paired observations and calculation produce Table 17? Can you reconcile them with Table 16?
- Does Figure 3 use the same model version, questions, and evaluation protocol as Table 3? What explains 66.1% versus 61.2%, and does the text-log comparison keep the model fixed?
- What audio, if any, was provided to each model for S10, and were the same questions and available information used in the human comparison?