acceptodds
Under review as a conference paper at ICLR 2027

VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis

Abstract

Spatio-temporal reasoning is a core capability for Multimodal Large Language Models (MLLMs) operating in the real world. As such, evaluating it precisely has become an essential challenge. However, existing spatio-temporal reasoning benchmark datasets primarily rely on static image sets or passively curated video data, which limits the evaluation of fine-grained reasoning capabilities. In this paper, we introduce VGenST-Bench, a video benchmark that employs generative models to actively synthesize highly controlled and diverse evaluation scenarios. To construct VGenST-Bench, we propose a multi-agent pipeline incorporating a human quality control stage, ensuring the quality of all generated videos and QA pairs. We establish a comprehensive video taxonomy, encompassing Spatial Scale, Perspective, and Scene Dynamics to span diverse scenarios. Furthermore, we design a hierarchical task suite that decouples low-level visual perception from high-level spatio-temporal reasoning. By shifting the paradigm from passive curation to active synthesis, VGenST-Bench enables fine-grained diagnosis of spatio-temporal understanding in MLLMs.

Primary area:
datasets and benchmarks
Keywords:
Spatio-Temporal Reasoning, Video Benchmark, Synthetic Benchmark, MLLM Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zURoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VGenST-Bench, a spatio-temporal reasoning benchmark built from generated videos. A multi-agent pipeline produces scene graphs, scenarios, videos, and QA pairs, and the scene specification serves as ground truth; two-stage human review filters the output. The benchmark covers a 3×2×2 taxonomy with one task per cell and three question levels, totaling 1,200 videos and 33K QA pairs (9,707 base MCQs). Across 15 MLLMs, accuracy drops sharply from L1 to L3, and supplying ground-truth facts as text raises L3 accuracy from 40.1% to 90.6%.

Critical strengths
  1. Controllable construction with useful diagnostics. The spec-derived ground truth allows balanced coverage of scenario types, and the "None-of-these answer" variant shows that models rank the given options rather than verify them against the video.
  2. Thorough, honestly scoped ablations: blind input (~1%), single frame, shuffled frames, text oracle, frame count, quality correlation, a realism study, and ranking agreement with real-video benchmarks.
Critical weaknesses
  1. Validity is only partially established. Ground truth depends on the generator faithfully following the spec, yet video QC agreement is moderate (κ=0.58). The real-video ranking correlation uses only 8 open-source models. With one task per taxonomy cell, axis effects are confounded with task difficulty.
  2. Some claims are under-supported. It is unclear whether humans saw full videos or the same 8 frames as the models. The paper attributes L3 failures to reasoning, yet the text oracle points to an extraction bottleneck. Novelty is moderate given prior generated-video benchmarks such as VideoHallu, and no confidence intervals are reported.
Questions for authors
  1. Did the human baseline use full videos or 8 frames?
  2. How often do retained videos deviate from the spec on the queried fact? Were regenerated videos re-reviewed?
  3. How specific are the text-oracle facts? Could they effectively reveal the answer?
  4. Does the ranking correlation hold with more models, including proprietary ones?
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We thank the reviewer for the constructive feedback and recognition of our controllable benchmark construction and diagnostic analyses. We address the two weaknesses below and the four questions in the following response.

zURo-W1&W2

Validity, interpretation, and scope of the benchmark.

Video QC agreement. The reported κ = 0.58 was computed as a pooled Cohen’s κ, which was inappropriate for our rotating-pair design. Recomputed as Fleiss’ κ, it remains 0.58 for the overall accept/reject decision, which combines physical plausibility, generation artifacts, and scenario adherence (t19V-Q3). For scenario adherence alone, which is more directly relevant to specification–video alignment, raw agreement is 96.8% and Fleiss’ κ is 0.75. Split cases were adjudicated blind to the identity of the rejecting annotator. These statistics describe agreement during initial screening, rather than the final benchmark’s answer-error rate. We address whether the final videos support the specification-derived answers in Q2.

Beyond the 8-model rank correlation. We conducted a matched real–generated study covering 50 scenarios (t19V-Q1). For each real 8-second video, we generated a counterpart conditioned on its first frame and the same scene specification, and evaluated identical questions at L1, L2, and L3. Across the eight open-source models, paired correctness agreement is 96.2% (95% CI: 95.2–97.3), and the L1-to-L3 accuracy drop is 46.5 pp on real videos versus 45.5 pp on generated videos. This provides complementary evidence that the level-wise decline also occurs in the matched real-video setting, within the scenarios and generation conditions studied. We address the model coverage of the ranking analysis separately in Q4.

One task per taxonomy cell. We agree that one task per cell prevents separating taxonomy-axis effects from task difficulty. The taxonomy organizes scenario construction across spatial scale, perspective, and scene dynamics; it does not establish exhaustive task coverage. We will clarify this scope and report axis-level scores as descriptive summaries of the included tasks.

Our diagnostic comparisons keep task identity fixed: frame shuffling, the text oracle, and answer-option variants change conditions for the same evaluation items. Their effects therefore cannot be explained by task differences across cells. Isolating axis effects would require multiple tasks per cell and matched tasks across axis conditions, which our benchmark does not provide.

Human input. Humans saw the same eight frames as the models, so the human–model comparison uses matched visual input. We clarify the evaluation protocol in Q1.

L3 failures: reasoning or extraction. We agree that the results should not be interpreted as isolating failures of abstract reasoning. Video-based spatio-temporal reasoning requires extracting and organizing visual evidence as well as applying reasoning operations to it. Providing ground-truth facts as text raises L3 accuracy from 40.1% to 90.6%, indicating a substantial contribution from visual extraction, grounding, and representation. Temporal organization also matters: shuffling reduces L3 accuracy by 10.7 pp, compared with 2.4 pp at L1. These results support complementary dependencies, rather than a clean separation of perception and reasoning. Q3 details the oracle inputs and addresses possible answer leakage.

Confidence intervals. We added 95% CIs using 2,000 video-level bootstrap resamples. Across all 15 models, overall accuracy CIs differ approximately 0.8–1.2 pp around each score. Human accuracy is 99.0% (95% CI: 98.7–99.3). On the 120-video ablation set, the L3 shuffle drop exceeds the L1 drop by 8.3 pp (95% CI: 4.1–12.5), and the L3 text-oracle gain is 50.5 pp (95% CI: 45.7–55.3). We will include these estimates and the resampling procedure in the revision.

Novelty relative to VideoHallu. We acknowledge VideoHallu as prior work evaluating MLLMs on generated videos. Its expert-crafted questions test detection of commonsense and physics violations. Our Video QC filters out such defects, and generated videos serve as controlled testbed for spatio-temporal reasoning. Scene graphs and scenarios guide video generation, answers derive from the generation specifications, and human review checks that the videos support them. Our contribution is this specification-driven construction, combined with hierarchical QA and diagnostics requiring ground truth independent of the video. We will clarify this distinction in the Related Work and narrow our novelty claims accordingly.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Thank you for your detailed questions about our benchmark work. We will address the following four questions.

zURo-Q1

Did the human baseline use full videos or 8 frames?

The same eight frames as the models. Participants viewed the eight uniformly sampled frames used for model evaluation, without access to scene specifications or reference answers, and were scored under the same circular evaluation. The 99.0% vs. 85.9% comparison therefore uses identical visual input. It also shows that the reference answers are recoverable from exactly the input the models receive. We will state this protocol explicitly in revision.

zURo-Q2

Deviation from the specification and re-review of regenerated videos.

Regenerated videos. The 115 videos rejected in the initial Video QC (70 rejected by both reviewers and 45 after adjudicating split cases) were regenerated from the same specifications. The authors re-reviewed each replacement against the original QC criteria: physical implausibility, generation artifacts, and prompt–video drift. This was not another round of independent paired Video QC, and we will state this in App. D.3. The replacements then went through QA QC, where two validators checked every answer against the video (9,800 MCQs in Table 13). All final videos, including replacements, were checked for prompt–video drift. The reported scenario-adherence κ = 0.75 pertains to the initial paired QC.

Deviation on the queried fact. The human baseline provides answer-level consistency evidence. Participants who saw only the eight evaluated frames of each final video, covering all 1,200 videos including the replacements, matched the specification-derived answers on 99.0% of questions under circular evaluation (L1 99.4%, L2 99.3%, L3 97.9%). These results support a high degree of consistency between human answers based on the evaluated frames and the specification-derived references.

However, these results do not tell us how often a video differs from its specification. A participant can make a mistake even when the video is correct, and a participant can still pick the reference answer from the options even when the video is slightly off. We therefore treat the human results as supporting evidence only. We have not directly measured how often the queried fact differs between the video and the specification.

zURo-Q3

Details of the text-oracle information.

We thank the reviewer for asking us to clarify the oracle inputs. In this condition, the frames are replaced by text. The model receives the ground-truth scene graph of the video, which describe objects, attributes (except answers). They are the same for every question on a video and are not selected per question. For example, for a PT–DE question, they dose not describe the direction of the start point from the final viewpoint, which must still be derived.

At the same time, this structured text can substantially simplify the questions, and the condition also changes the input modality from video to text. The increase in L3 accuracy from 40.1% to 90.6% therefore indicates that obtaining usable spatio-temporal information from video is an important bottleneck, but it does not cleanly isolate reasoning ability. We will revise the related statements in revision. We also found that App. E currently describes text oracle inputs as scene graph and scenario, but the input is the scene graph only, so we will correct this wording.

zURo-Q4

Does ranking agreement extend to more models, including proprietary ones?

We thank the reviewer for asking whether the ranking agreement extends to a broader model set. Our current cross-benchmark comparison covers eight open-source MLLMs, yielding Spearman correlations of 0.905 with VSI-Bench and 0.929 with EgoExoBench. These results establish strong ranking agreement within this model set.

To strengthen the validity evidence beyond model rankings, we additionally conducted the matched real–generated study (t19V-Q1). This study compares identical questions and answer options across 50 paired scenarios. Across the eight evaluated models, correctness agreement is 96.2% (95% CI: 95.2–97.3), and the L1-to-L3 accuracy decline is similar on real and generated videos (46.5 and 45.5 pp). At L3, paired correctness agreement is 94.8%. The high correctness agreement therefore shows that, within the tested scenarios, models tend to succeed or fail on the same questions across the two real or generated video sources.

These analyses address complementary aspects of validity. Cross-benchmark correlations assess consistency in relative model performance, while the matched study examines consistency at the question level under controlled scenarios. We will therefore report both forms of evidence in the revision, explicitly scope the ranking results to the evaluated open-source models, and identify extension to proprietary models as an open validation question.

Reviewer_UaguOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents VGenST-Bench, a benchmark for evaluating spatio-temporal reasoning in MLLMs through actively generated videos. A multi-agent pipeline produces scene graphs, event scenarios, videos, and QA pairs, followed by human quality control. The benchmark contains 1,200 videos across 12 tasks organized by spatial scale, perspective, and scene dynamics. Questions span visual perception, scene understanding, and spatio-temporal reasoning, with multiple-choice and reformulated variants. Experiments on 15 MLLMs show lower accuracy at higher question levels and sensitivity to answer formats and temporal order. Human baselines, text-oracle tests, video-quality analyses, and comparisons with real-video benchmarks further examine the benchmark’s validity and the sources of model errors.

Critical strengths
  • Using video generation to construct targeted reasoning scenarios is an interesting direction. Specifying spatial relations and events before generating videos offers useful control over cases that are difficult to collect from real data.
  • The paper transforms this idea into a concrete benchmark with 1,200 videos and a structured construction pipeline. The scene specifications, QA design, and human verification provide a useful basis for further work.
  • The evaluation goes beyond a model leaderboard. Human baselines, frame-order ablations, text-oracle tests, and comparisons with real-video benchmarks provide useful evidence and diagnostic insights.
Critical weaknesses
  • It is unclear whether the three question levels measure different reasoning abilities. A question about a hypothetical situation or a new viewpoint may still be answered from a single frame or general knowledge. More direct checks are needed to show that L3 questions actually require harder reasoning.
  • The benchmark has many visual themes, but the range of tasks is still limited. Each combination of scale, perspective, and scene dynamics contains only one task. Changing the setting does not necessarily change the reasoning needed. It is therefore difficult to tell whether score differences come from the scene conditions or the tasks themselves.
  • More evidence is needed that the generated videos support the intended answers. Human checks and VBench scores are helpful, but they do not fully show that objects, movements, and events follow the original scene descriptions. The human quality study also uses only eight sampled frames. The paper should also clarify whether regenerated videos went through the same quality checks.
Questions for authors
  • How do the authors ensure that L3 questions require inference beyond direct observation?
  • Were regenerated videos checked again under the same QC protocol? Can the authors report agreement between answers derived independently from the final videos and those derived from the scene specifications?

Additional Comments

The novelty claim should be stated more narrowly in relation to, e.g., VF-Eval (ACL 2025), which already evaluate MLLMs on generated videos, including spatial or temporal understanding. The more distinctive contribution here appears to be scene specification-driven construction for dedicated spatio-temporal evaluation.

Figures 2, 4, and 7 would benefit from more informative content.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We appreciate the reviewer's recognition of the potential of video generation for constructing targeted reasoning scenarios, the concrete benchmark construction pipeline, and the diagnostic value of our evaluation. We address the concerns in four parts (Q1–Q4).

Uagu-Q1

Validity of the L3 question design and its inference requirements.

We thank the reviewer for highlighting the need to make the inference requirements of L3 questions explicit. We agree that hypothetical wording or a novel viewpoint formulation alone does not establish that a question requires inference beyond direct observation.

Three L3 question types and their required operations. Our L1–L3 design distinguishes the operations targeted by each question: recognizing visible entities and attributes (L1), integrating observations into spatial and temporal relationships (L2), and transforming these relationships under a specified viewpoint, altered condition, or continuation assumption (L3). Below, we clarify this distinction through concrete benchmark L3 questions and examine potential shortcuts using our input ablations.

Table R10 presents actual PT–DE, CR–RV, and PR–QC questions.

L3 questionRecorded answerRequired operation
PT–DE: “At the final position, in which direction (using clock positions where 12 o'clock is straight ahead) is the starting point located?”7–8 o'clock (back-left)Retain the start position, integrate the observed movement and turn, and transform the start-to-final relation into the final observer's reference frame.
CR–RV: “If the Prototype Racer had been moving slower than the camera object, what would happen in the video?”The Prototype Racer would not appearPreserve the racer's initial behind-camera position and the forward-motion setting, change its speed relative to the camera, and infer that it would no longer catch up and enter the forward view.
PR–QC: “If the hand were to perform one additional ADD action after the video ends, how many items would be inside the container?”2Recover the final count from ADD, ADD, REMOVE starting with an empty tray, then apply the specified continuation: one item becomes two.

Table R10. Actual L3 questions, recorded answers, and required operation.

These examples illustrate the operations targeted by L3. Fig. 7 provides a concrete PT example. The model's response correctly describes key observed events but gives an incorrect relative direction. This illustrates the distinction between describing observations and applying a reference-frame transformation. We next examine visual and temporal dependence through the existing input ablations.

Existing ablations test visual and temporal dependence. App E.5 evaluates four models on the same 120-video subset. The questions and scoring protocol remain fixed, while the visual input changes. Table R11 summarizes the existing results.

Input conditionL1L2L3
Blind1.11.11.2
Single random frame31.326.725.2
Original (K = 8)84.662.640.1
Shuffled frames (K = 8)82.257.129.4
Drop after shuffling, pp2.45.510.7

Table R11. Existing input ablation results from Table 19: accuracy (%) under circular evaluation, averaged over four models on a 120-video subset.

Text-only shortcuts. Without video, accuracy is 1.1–1.2% across levels, close to the 0.8% random baseline under circular evaluation. These results show limited success from question wording and answer options alone for the tested models.

Single-frame cues. At L1, accuracy rises from 31.3% with one random frame to 82.2% with eight shuffled frames, close to the 84.6% obtained in the original order. At L3, the corresponding scores are 25.2%, 29.4%, and 40.1%. For L3, the gain over a single frame is 4.2 pp with eight shuffled frames and 14.9 pp with eight ordered frames. Thus, presenting these eight frames in their original order provides an additional 10.7 pp improvement over shuffling them.

Temporal order. Shuffling preserves the sampled frame set while changing its order. The resulting decline is 10.7 pp at L3, compared with 5.5 pp at L2 and 2.4 pp at L1. The larger observed L3 drop supports substantial sensitivity to temporal order.

Additionally, two validators independently reviewed every base MCQ against its video, and the rejection criteria included (iii) single-frame solvability for L3 questions and (iv) language-prior shortcuts solvable without the video (App. D.3).

Together, our L3 design targets viewpoint transformation, reasoning under changed conditions, and prediction from observed scene information. The examples clarify these intended operations. We will report these results in the revision.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Uagu-Q2

Alignment between scene specifications, final videos, and answers.

We thank the reviewer for asking us to clarify the quality control applied to regenerated videos and to provide more direct evidence that the final videos support the specification-derived answers.

Regenerated videos and follow-up review. In the initial Video QC round, 70 videos were rejected by both reviewers. Among the 85 split cases, author review accepted 40 and rejected 45. The 115 rejected cases were regenerated using the same scene specifications. Before inclusion in the final benchmark, the authors reviewed each replacement using the original Video QC criteria: (i) physical implausibility, (ii) generation artifacts, and (iii) prompt–video drift.

The authors handled regeneration and the follow-up video checks. The replacements then went through QA QC, where two validators checked every answer against the video (9,800 MCQs in Table 13). On scenario adherence alone, the initial paired Video QC reached 96.8% raw agreement (Fleiss' κ = 0.75). We will clarify this in the revision.

Agreement between video-based answers and specification-derived answers. Our existing human baseline provides an answer-level comparison. Participants answered the questions using only the eight frames of each video, without access to the scene specifications or answers. Their responses were compared with the answers under the same circular evaluation protocol used for the models. Human accuracy was 99.0% overall, with 99.4% at L1, 99.3% at L2, and 97.9% at L3.

Each of the 1,200 final videos was answered by one participant, so this check covers the whole benchmark. The result shows that human answers from the eight frames match the reference answers in almost all cases. However, it does not tell us how often a video differs from its specification because participants saw only eight frames and chose from given options.

Scope of the eight-frame video quality study. The video quality study in App. E.6 compares the perceptual quality of our videos with real-video benchmarks, using eight frames sampled with the same uniform protocol as model evaluation. We agree that this study cannot establish that full videos follow the scene specifications. It was designed to assess perceptual quality, not specification adherence. Specification adherence is instead checked on complete videos. Each full video was inspected for prompt–video drift (by the validator pairs, or by the authors for replacements), and QA QC validators checked each ground-truth answer against its video.

Uagu-Q3

Task coverage and interpretation of results across the taxonomy.

We agree that visual theme diversity does not establish task diversity, and that one task per cell cannot separate scene-condition effects from task difficulty. Our taxonomy guides the construction of scenarios across spatial scales, perspectives, and scene dynamics. Covering these combinations does not imply complete coverage of all reasoning tasks.

This limitation affects comparisons across taxonomy cells. Our controlled diagnostic experiments, however, keep the underlying tasks fixed. For example, the input ablation in Table 19 uses the same questions and 120 videos across conditions. Shuffling the same eight frames reduces L3 accuracy from 40.1% to 29.4%. Similarly, the None-of-these variants modify answer options for the same underlying videos and questions. These results provide evidence of sensitivity to temporal order and answer format without changing task identity. We will make this distinction explicit, narrow claims of comprehensive task coverage, and describe axis-level scores as comparisons among the included tasks.

Uagu-Q4

Novelty relative to VF-Eval

We thank the reviewer for pointing out the comparison to VF-Eval. Regarding novelty, we agree that VF-Eval already evaluates MLLMs on generated videos, including spatial and temporal reasoning. The two benchmarks use generated videos differently. VF-Eval evaluates whether MLLMs can provide reliable feedback on AIGC videos, through coherence validation, error awareness, error type detection, and reasoning evaluation. In VGenST-Bench, generated videos serve as controlled stimuli for spatio-temporal reasoning. Scene graphs define basic scene info such as object relations, scenarios specify events and camera motion, and the answers are derived from these specifications and then verified by humans. Because the ground truth exists as a symbolic state independent of the video, we can run diagnostics that require it, such as the text oracle, which replaces the frames with the scene graph (L3: 40.1% -> 90.6%). We will cite VF-Eval and narrow the “first” statements to specification-driven construction for spatio-temporal reasoning evaluation.

Reviewer_UaguOpenReview ↗

Official Comment by Reviewer Uagu

I thank the authors for the clarifications, especially regarding the human evaluation protocol and quality checks for regenerated videos. These address much of my concern about whether the generated videos align with the reference answers. I therefore raise my rating.

However, the L3 validation is still limited. A model may fail with a randomly chosen frame but answer correctly if shown the right frame. A more careful check of such cases would strengthen the benchmark.

Reviewer_t19VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VGenST-Bench is a video benchmark for spatio-temporal reasoning in MLLMs whose videos are synthesized by video generative models rather than curated from the web or 3D scene datasets. A four-agent pipeline (scene graph → scenario → video → QA, each generator paired with an LLM validator) produces items, followed by two-stage human QC on videos and QA. Videos follow a 3×2×2 taxonomy (spatial scale × ego/exo perspective × static/dynamic), with one reasoning task per cell: 12 tasks, 100 videos each (1,200 total). Orthogonally, each video carries QA from 12 types across three levels (L1 perception, L2 scene understanding, L3 spatio-temporal reasoning), gated by a task–QA applicability matrix; 9,707 base MCQs are expanded into three variants (None-of-these distractor, None-of-these answer, open-ended) for 33K QA pairs.

Fifteen MLLMs are evaluated under circular evaluation. Humans reach 99.0% vs. 85.9% for the best model, and all models degrade sharply from L1 to L3 while humans drop 1.5 points. Supporting analyses: rank correlation with VSI-Bench/EgoExoBench (ρ = 0.905/0.929 over eight open-source models), weak association between VBench quality proxies and accuracy, frame-sampling and input ablations (shuffling frames costs 10.7 points at L3; a ground-truth text oracle lifts L3 from 40.1% to 90.6%), and a blind human study where the clips rank worst on photorealism but closest to real video on scene comprehensibility.

Critical strengths

S1. The V1/V2 asymmetry is a finding that outlives this benchmark. Adding "None of these" as an extra distractor barely changes accuracy, but replacing the correct option with it collapses accuracy across every model tested. That dissociation is clean evidence that models rank the given options against one another rather than verifying an answer against the video — so closed-form MCQ accuracy, the format most video-reasoning leaderboards rely on, systematically overstates competence. This drives my recommendation because it is a reusable, cheap-to-adopt protocol rather than a property of this particular dataset.

S2. Active synthesis enables a diagnostic that no curated benchmark can run. Because ground truth comes from the scene graph that drives generation, the authors can swap the video for the ground-truth symbolic state: L3 accuracy goes from 40.1% to 90.6%. Together with the shuffled-frame result (−10.7 points at L3 vs. −2.4 at L1), this localizes the failure to extracting a usable spatio-temporal representation from video rather than to downstream inference. It is consequential because it converts the paper from a leaderboard into an actual diagnosis, and it is the concrete payoff of the synthesis paradigm the paper argues for.

Critical weaknesses

W1. The core validity claim rests on one weak number. The only evidence that synthesized video measures real spatio-temporal reasoning is Spearman ρ = 0.905 / 0.929 against VSI-Bench / EgoExoBench over eight models. At n = 8 across a wide capability range, almost any multimodal benchmark would rank those models similarly, since the dominant varianceis general model strength. The decisive experiment — identical scenarios as both generated and real video, compared per item — is absent, so I cannot rule out that part of the L1→L3 gap reflects synthetic-distribution artifacts rather than reasoning. This is the paper's thesis, so it determines my score.

W2. Ground truth is anchored to the generation spec, and the check that the video realizes that spec is the weakest link. Correctness comes from the scene graph, so prompt–video drift silently invalidates items. Yet Stage-1 video QC reports Cohen's κ = 0.58 — moderate agreement on precisely the judgment "does this clip instantiate the intended scenario" — with 85 split cases resolved by the non-blind authors. This places an unquantified error floor under the 99.0% human ceiling and every level-wise gap the paper's diagnoses are built on.

Additional concerns (serious but not decisive):

  • W3. The ~23.7K reformulation variants got no human verification. This matters most for V2, which presupposes no remaining distractor is correct for that specific video — unchecked, yet V2 carries the paper's strongest conclusion (my S1).
  • W4. Stage-2 rejects 93 of 9,800 MCQs (0.9%) — hard to distinguish from LLM validators confirming the priors of an LLM generator conditioned on the same scene graph. The QA generator and the open-ended judge are also both Claude-Sonnet-4.6.
  • W5. 48% of videos come from a single generator (seedance-2.0-fast), so "10 generators" is nominal; all ablations use only 120 videos and four small open-source models; no seeds, variance, or error bars anywhere.
  • W6. The human baseline has one annotator per video with no inter-human agreement reported, so the headline 13-point model–human gap has no error bars.
Questions for authors
  1. Validity (W1). Can you provide even a small matched-pair study — 50–100 scenarios instantiated as both generated and real/recorded video with identical QA — reporting per-item agreement and per-level accuracy on both? If infeasible, can you show your rank correlation is not merely recovering general model strength (e.g., ρ restricted to models of comparable scale, or partial correlation controlling for a general-capability score)?
  2. V2 verification (W3). For a random sample of ~300 V2 items, what fraction has a remaining distractor that is in fact correct or defensible for that specific video? This rate directly bounds your strongest conclusion.
  3. QC reliability (W2). What is κ for the scenario-adherence criterion alone, separated from artifact-based rejections? How many of the 85 splits were adherence disagreements, and were the author resolvers blind to which annotator rejected?
  4. Generator confound (W5). After conditioning on task, does accuracy differ across source generators (e.g., seedance-2.0-fast vs. veo-3.1-fast)? A generator main effect would mean scores partly track generator idiosyncrasies.
  5. Human baseline (W6). Can you report inter-human agreement on an overlapping subset and per-task human CIs? Also, did participants really answer all N cyclic permutations of every question (Sec. 4.1) — that is ~10³ items per annotator — and how were ordering and fatigue controlled?
  6. Judge circularity (W4). What is the agreement between your Claude-Sonnet-4.6 open-ended judge and human judgments on a sampled subset, and does the open-ended L3 drop survive a judge from a different model family?
  7. Reporting. Table 11 sums to 1,085/1,200 (90.4%) while Table 13 reports 1,045 both-pass (87.1%) plus 40 author-accepted splits. Please state explicitly that 90.4% is post-author-resolution; as written the two read as inconsistent.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/4)

We thank the reviewer's recognition of the V1/V2 reformulations as a reusable diagnostic of MLLMs reliance on answer choices. We also appreciate the positive assessment of our text-oracle and frame-order analyses, which help examine how models use scene information and temporal order. We address the questions below and conducted additional experiments to support our claims.

t19V-Q1

Construct validity of the benchmark.

We agree that model-level rank correlation alone cannot resolve this validity question. Following your valuable suggestion, we constructed 50 matched real–generated video pairs (100 videos) representing the same scenarios.

(a) Matched comparison of the L1-to-L3 drop. We use 50 scenarios from MC, QC, CI, IO, and HO tasks. Each real clip is an 8s video captured by the authors, and its first frame conditions a new 8s generated video following the same scenario specification. For each scenario, we construct one question at each of L1, L2, and L3. We use identical questions and answer options for both video versions.

For this comparison, we evaluated eight open-source MLLMs, the same models used in App E.1. The results are summarized in Table R1.

Video sourceL1L2L3Gap (L1-L3)
Real video87.566.841.046.5
Generated video86.865.541.345.5

Table R1. Model accuracy(%) by reasoning level and video source, averaged across eight models. Gap is L1-L3, in pp.

Accuracy declines from L1 to L3 for both sources. L1 accuracy is 87.5% on real videos and 86.8% on generated videos, while L3 accuracy is 41.0% and 41.3%, respectively. The mean L1-to-L3 drops are 46.5 pp for real videos and 45.5 pp for generated videos. Thus, this result shows a similarly large level-wise decline in both.

(b) Paired item-level agreement and confidence intervals. We compare accuracy on the same question across each video pair. Table R2 summarizes the outcomes.

Paired outcomeProportion (%)
Both correct62.9
Real only correct2.2
Generated only correct1.6
Both incorrect33.3
Overall correctness agreement96.2 (95% CI: 95.2–97.3)

Table R2. Paired outcomes on matched real–generated videos. The 95% CI is computed from 10,000 task-stratified paired bootstrap resamples of the 50 scenario pairs.

The models agree in correctness on 96.2% of paired evaluations, with only 3.8% yielding different correctness outcomes across the two video versions. At L3 specifically, correctness agreement is 94.8%. In 56.3% of all paired L3 evaluations, the model answers both the real-video and generated-video versions incorrectly.

Together, these results provide direct evidence beyond cross-model rank correlation. The same models exhibit both a comparably large L1-to-L3 decline and high item-level correctness agreement across real and generated videos. We interpret these findings as indicating that, for matched scenarios, the L1-to-L3 decline is not specific to generated videos. It carries over to real videos of the similar scenarios, and models largely fail the same L3 items on both sources.

t19V-Q2

Validity of the V2 reformulations.

Thank you for highlighting the need to verify whether any remaining V2 distractor is correct or defensible for the actual video.

We randomly sampled and audited 300 QAs from the full V2 set. Two annotators independently classified each item into one of three categories. (A) valid, with all remaining non-“None-of-these” options incorrect; (B) containing at least one defensible remaining option; or (C) insufficiently supported by the video to determine the answer. Table R3 summarizes the outcomes.

Audit outcomeCount
(A) Valid V2 item298
(B) Correct or defensible remaining option2
(C) Insufficient video evidence0

Table R3. Audit outcomes on 300 V2 QAs.

Only two flagged QAs primarily concerned color-related questions with borderline interpretations of the answer options. Both involve color attributes, where a near-miss distractor (e.g., an adjacent color term) becomes arguably defensible once the correct option is removed.

This result is consistent with our construction. The 9,707 base MCQs underwent human QC that explicitly checked for incorrect ground-truth answers and ambiguous or multiply-correct options (App D.3), and V2 replaces only the correct option with “None of these” while preserving the question and the original distractors. As a result, the remaining distractors stay incorrect in nearly all cases.

We hope this additional experiment addresses the reviewer's concern about defensible remaining options. The result provides empirical support for the validity of the V2 reformulations, while identifying a very small number of residual ambiguities.

AuthorsOpenReview ↗

Rebuttal by Authors (2/4)

t19V-Q3

Reliability of video QC and scenario–video alignment.

We thank the reviewer for closely examining our video QC process and highlighting the need to verify that the scenario derived ground truth holds in the generated videos. We acknowledge that our description of the video QC procedure and the reported inter-annotator agreement was not sufficiently clear.

Below, we first clarify the QC procedure and the agreement statistics reporting, and then address the three questions in turn.

Clarified procedure. Twelve validators reviewed the videos in a rotating-pair design. The overall accept/reject decision combined three rejection criteria. (i) physical implausibility, (ii) generation artifacts, and (iii) prompt–video drift (Scenario adherence). For 1,200 videos, 1,045 (87.1%) were accepted by both annotators, 70 (5.8%) were rejected by both, and 85 (7.1%) were split decisions.

We inspected these 85 cases, accepting 40 and rejecting 45. Consequently, 1,085/1,200 videos (90.4%) were retained after author resolution. We will correct the wording “90.4% pass both reviews” in App D.3 to distinguish the independent both-pass rate (87.1%) from the post-resolution retention rate (90.4%). We will also revise the description of the decision rule ("intersection rule") in Sec. 3.3 and App. D.3 to match the actual procedure. Videos accepted by both validators are retained, split cases are adjudicated by the authors, and rejected videos are regenerated and re-reviewed by the authors before the QA QC.

Agreement statistic. In the original paper, we computed the reported Cohen's κ from a pooled 2x2 table, treating the two annotators within each task as “Annotator A” and “Annotator B.” However, we noticed that this metric is not applicable since these roles corresponded to different individuals. To summarize agreement across this rotating-pair design, we will replace the pooled Cohen's κ with Fleiss' κ.

Using the independent paired judgment counts in Table 13, the recalculated Fleiss' κ is 0.583, which rounds to 0.58. Raw agreement is (1,045 + 70)/1,200 = 92.9%. These statistics are based on the original independent judgments before author resolution. We will update the agreement analysis and its description in App D.3.

(a) Agreement on scenario adherence alone.

To assess scenario adherence separately, we classified each of the 225 rejection decisions across the 155 videos receiving at least one rejection. Both-reject videos contribute two reasons each (70 × 2), and split videos contribute one (85). The 1,045 original both-accept videos remain both accept for adherence. Table R4 summarizes the result across all 1,200 videos.

Scenario-adherence judgmentVideos
Both accept1,096
Both reject65
Disagreement39
Total1,200

Table R4. Agreement on scenario adherence across all 1,200 videos before author resolution.

The raw agreement for scenario adherence was 96.8%, and Fleiss'κ was 0.75, using the original independent judgments before author resolution. The 39 disagreements comprise 36 original split cases and three original both-reject cases in which only one annotator cited drift. This indicates greater agreement on whether videos matched the scenarios used to define the ground-truth answers.

(b) Scenario-adherence disagreements among the 85 split cases.

For 85 accept/reject disagreements, 36 involved scenario adherence and 49 involved only physical implausibility or generation artifacts. Table R5 summarizes the results.

Source of disagreementVideosAuthor-acceptedAuthor-rejected
Scenario adherence involved36333
Other criteria only493712
Total854045

Table R5. 85 split videos decisions and their resolution.

36 of the 85 splits were adherence disagreements. The remaining 49 involved only physical implausibility or generation artifacts. We rejected 33 of these 36 adherence splits and accepted three.

(c) Blinding during author resolution.

We inspected all 85 split videos while blinded to which annotator had accepted or rejected each clip. Final decisions were based on the video evidence and the QC criteria, without relying on the identity or perceived reliability of individual annotators. We will explicitly document this blinding procedure in App D.3.

Together, these results show that our Video QC process combines independent video screening and blinded adjudication of disagreements. The overall κ summarizes agreement on the initial screening decisions. It does not estimate the ground-truth error rate of the final benchmark. These complementary checks support the reliability of our benchmark construction by examining whether the generated videos support the intended answers.

AuthorsOpenReview ↗

Rebuttal by Authors (3/4)

t19V-Q4

Generator concentration, statistical uncertainty, and ablation scale.

We address generator imbalance and statistical uncertainty through task-conditioned comparisons, generator exclusion, and bootstrap CIs.

(a) Task-conditioned generator comparison. Generators are unevenly distributed across tasks, so we compare generators within tasks. For each contrast, we keep only the tasks in which each group has at least 10 videos. For every MLLM, we compute the accuracy difference between the two groups within each task (circular evaluation, base MCQs) and average these differences across tasks. In the "all others" contrast, all videos not produced by seedance-2.0-fast are pooled into one group.

ContrastMean within-task ΔRange across models
All others − seedance-2.0-fast−0.6−2.6 to +0.7
wan-2.6 − seedance-2.0-fast−0.6−2.6 to +1.2
veo-3.1-fast − seedance-2.0-fast−0.2−4.8 to +2.6

Table R6. Task-conditioned accuracy differences, averaged and ranged over the 15 models (pp).

After conditioning on task, the average accuracy differences between generator groups are small. The absolute mean difference across the 15 models is below 1 pp for every contrast, and the direction of the differences varies across models. We therefore do not observe a consistent accuracy advantage for any compared generator group.

(b) Main findings without seedance-2.0-fast. We recompute the level-wise results with all 586 seedance-2.0-fast videos removed. For comparison, we evaluate both the original and reduced datasets on the same 11 tasks, excluding LS, with equal weight assigned to each task.

Averaged across the 15 MLLMs, L1/L2/L3 accuracy changes from 88.1/68.3/45.8% to 87.9/68.1/45.8%. The mean L1-to-L3 gap changes from 42.3 to 42.1 pp, and the maximum absolute change in any model's gap is 3.3 pp. Model rankings also remain highly similar (Spearman ρ = 0.996).

These results show that the pronounced L1-to-L3 decline and model rankings are largely preserved after excluding the dominant generator within the shared 11-task coverage.

(c) Uncertainty and ablation scale. We added 95% CIs from 2,000 video-level bootstrap resamples. We resample videos because questions on the same video are correlated. For overall accuracy, the half-widths are 0.8–1.2 pp for all 15 models. Applying the same paired bootstrap to the 120-video ablations gives the following. The L3 shuffle drop exceeds the L1 drop by 8.3 pp (95% CI: 4.1–12.5) and is positive for all four models. The text-oracle gain at L3 is +50.5 pp (95% CI: 45.7–55.3).

Together, the task conditioned comparison and the generator exclusion analysis indicate that our main results do not depend on the dominant generator, and the video-level bootstrap CIs quantify sampling uncertainty in accuracy estimates and support the frame-shuffling and text-oracle effects. We will include these analyses and uncertainty estimates in the revised manuscript.

t19V-Q5

Human baseline reliability and workload.

We clarify the human evaluation protocol and assess participant variability and per-task uncertainty using two-stage bootstrap CIs.

Protocol and workload. Ten participants directly answered every cyclic permutation through a web-based interface. As described in Sec. 4.1 and App. F, each of the 10 participants evaluated 120 videos, with 10 videos per task. Evaluation was self-paced. A question was counted as correct only when all permutations were answered correctly, using the same scoring rule as for models. Self-paced participation does not, however, establish the absence of fatigue or order effects. We will clarify this procedure in the revision.

Human baseline details. Participant macro accuracies range from 98.59–99.43%. We ran 2,000 two-stage bootstrap resamples, first participants, then videos within each sampled participant and task, retaining all questions per video. Accuracy is macro-averaged across QA types, then tasks, as in Table 4. Overall accuracy is 99.00% (95% CI: 98.74–99.25).

TaskAccuracy [95% CI]
MC100.00 [100.00–100.00]
QC100.00 [100.00–100.00]
CI100.00 [100.00–100.00]
CM99.50 [98.86–100.00]
DE98.00 [96.71–99.00]
IO99.50 [98.75–100.00]
HO100.00 [100.00–100.00]
VI96.99 [95.26–98.70]
DS100.00 [100.00–100.00]
RV99.00 [98.00–99.75]
LS96.08 [94.43–97.52]
BT98.97 [97.84–99.75]

Table R7. Per-task results (%).

These results show that the human average remains near ceiling under resampling of the observed data. However, each video was answered by only one participant, so inter-human agreement remains unmeasured and participant effects cannot be separated from video difficulty. We will report these estimates and clarify the evaluation procedure and limitations in the revised manuscript.

AuthorsOpenReview ↗

Rebuttal by Authors (4/4)

t19V-Q6

Open-ended judge validation and cross-family robustness.

We thank the reviewer for raising judge circularity. We compare Claude's judgments with those of two authors and examine sensitivity to a judge from another model family.

Human validation. We randomly sample 40 V3 responses per level from each of the five models in Fig. 6(b), totaling 600 responses. Two authors (Author1 and Author2) each grade the same sample using same condition as LLM-as-Judge. Both are blinded to Claude judgments. Table R8 reports the results on this shared sample.

EvaluatorL1L2L3L1−L3 gapAgreement with Claude
Claude-Sonnet-4.662.547.034.528.0—
Author161.048.534.027.095.8
Author261.548.033.528.096.0

Table R8. Accuracies and agreement (%). L1-L3 gaps in pp. Each level contains 200 responses; agreement counts are 575/600 and 576/600. Sample scores need not equal full-set scores.

Human scores differ from Claude scores by 0.5–1.5 pp across levels, while the L1-to-L3 gap changes by at most 1.0 pp. Thus, the level-wise decline persists under both authors' judgments on this sample. This audit provides a consistency check on Claude's grading.

Alternative judge. We use DeepSeek V4.1 Flash to re-grade the same 600 V3 responses used for human validation in Table R8. The sample contains 40 responses per model and level across the five models in Fig. 6(b). We change only the judge. Table R9 compares level-wise accuracies and L1−L3 gaps on this shared sample.

JudgeL1L2L3L1−L3 gap
Claude-Sonnet-4.662.547.034.528.0
DeepSeek V4.1 Flash61.046.533.527.5

Table R9. Accuracies (%) and L1−L3 gaps (pp) on the same 600 sampled responses, with 200 responses per level.

In this comparison, changing the judge shifts L1/L2/L3 accuracy by at most 1.5 pp, and reduces the L1−L3 gap by 0.5 pp. The level-wise decline remains substantial under both judges.

Together, these results show the observed decline not depends on Claude's grading. On the shared 600-response subsample, accuracy and the L1-to-L3 gap change only slightly across evaluators. The L1-to-L3 decline persists under both author based and alternative model grading, supporting its robustness to judge choice within this sample.

t19V-Q7

Reporting clarification.

Thank you for pointing out this imprecise wording. As clarified in our response to Q3, 1,045/1,200 videos (87.1%) passed both independent reviews. Author resolution accepted 40 of the 85 split cases, yielding 1,085/1,200 (90.4%). Thus, the 90.4% in Table 11 is the post-author-resolution retention rate. We will correct "90.4% pass both reviews" in App. D.3, revise the decision-rule description in revision.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We sincerely thank the reviewers for their careful reading and for recognizing our controllable, specification-driven construction, the question reformulations as a reusable diagnostic, and our diagnostic evaluation beyond leaderboard scores.

In this rebuttal, we addressed the following shared concerns. Details and tables are in the per-reviewer responses.

1. Validity of generated videos as a testbed

  • Matched real–generated study (t19V, zURo). On 50 scenarios with matched real and generated videos and identical questions, paired correctness agreement is 96.2%. The L1-to-L3 drop is 46.5 pp on real videos versus 45.5 pp on generated videos.
  • Generator concentration (t19V). Within each task, accuracy differences between generators are below 1 pp. Removing the dominant generator preserves the L1-to-L3 gap and model rankings (ρ = 0.996).

2. Alignment between specifications, videos, and answers

  • Video QC reliability (t19V, zURo). For scenario adherence alone, raw agreement is 96.8% and Fleiss’ κ is 0.75. The authors resolved split cases without knowing which annotator had rejected the video.
  • Final videos and answers (Uagu, zURo). The authors re-reviewed regenerated videos using the original criteria, without a second round of independent paired QC. These videos then went through QA QC. Humans who saw the same eight frames as the models matched the specification-derived answers on 99.0% of questions. This is supporting evidence only. We have not directly measured how often a video differs from its specification on the queried fact.
  • V2 reformulations (t19V). In an audit of 300 V2 items, 298 were valid.

3. What the L3 results show

  • L3 inference requirements (Uagu). Blind accuracy is near chance. A single frame yields 25.2% accuracy at L3, compared with 40.1% with eight ordered frames. Shuffling reduces accuracy by 10.7 pp at L3 versus 2.4 pp at L1. These results show that L3 accuracy depends on multiple frames and their order. They do not show that every L3 question is unsolvable from a single frame.
  • Reasoning versus extraction (zURo, t19V). The text oracle raises L3 accuracy from 40.1% to 90.6%, suggesting that extracting usable spatio-temporal information from video is a major bottleneck. The oracle uses the scene graph with the answers excluded. However, structured text can simplify the questions, and this condition also changes the input modality. It therefore does not cleanly isolate reasoning ability. We will revise the related statements accordingly.

On robustness and scope (t19V, zURo, Uagu). We computed 95% bootstrap confidence intervals and will report them in the revision. The decline in accuracy across levels also holds under human and cross-family judges. We will also narrow our claims on taxonomy axis effects, ranking agreement, and novelty relative to VideoHallu and VF-Eval.

We thank the reviewers again for their constructive feedback. The discussion helped us provide stronger evidence for our benchmark’s validity and make our claims more precise. We will include all of these results and clarifications in the revised manuscript and would be happy to provide further analysis during the discussion period.

AuthorsOpenReview ↗

Revision update

We thank the reviewers for constructive feedback and thoughtful suggestions.

We revised the manuscript to incorporate the additional experiments and clarifications presented in our rebuttal. Key revisions are highlighted in blue. Detailed results and explanations are provided in the revised manuscript and our responses to each reviewer.

We appreciate for the reviewers and consideration and welcome any further questions.

AuthorsOpenReview ↗

Gentle follow-up on our rebuttal

Dear Reviewers,

Thank you again for your time and thoughtful feedback. We have posted our detailed responses and uploaded a revised manuscript with additional experiments and clarifications.

As the discussion period went to close, we would appreciate any follow-up feedback, whether our responses address your main concerns. We would be happy to provide any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.