- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces VGenST-Bench, a spatio-temporal reasoning benchmark built from generated videos. A multi-agent pipeline produces scene graphs, scenarios, videos, and QA pairs, and the scene specification serves as ground truth; two-stage human review filters the output. The benchmark covers a 3×2×2 taxonomy with one task per cell and three question levels, totaling 1,200 videos and 33K QA pairs (9,707 base MCQs). Across 15 MLLMs, accuracy drops sharply from L1 to L3, and supplying ground-truth facts as text raises L3 accuracy from 40.1% to 90.6%.
Critical strengths
- Controllable construction with useful diagnostics. The spec-derived ground truth allows balanced coverage of scenario types, and the "None-of-these answer" variant shows that models rank the given options rather than verify them against the video.
- Thorough, honestly scoped ablations: blind input (~1%), single frame, shuffled frames, text oracle, frame count, quality correlation, a realism study, and ranking agreement with real-video benchmarks.
Critical weaknesses
- Validity is only partially established. Ground truth depends on the generator faithfully following the spec, yet video QC agreement is moderate (κ=0.58). The real-video ranking correlation uses only 8 open-source models. With one task per taxonomy cell, axis effects are confounded with task difficulty.
- Some claims are under-supported. It is unclear whether humans saw full videos or the same 8 frames as the models. The paper attributes L3 failures to reasoning, yet the text oracle points to an extraction bottleneck. Novelty is moderate given prior generated-video benchmarks such as VideoHallu, and no confidence intervals are reported.
Questions for authors
- Did the human baseline use full videos or 8 frames?
- How often do retained videos deviate from the spec on the queried fact? Were regenerated videos re-reviewed?
- How specific are the text-oracle facts? Could they effectively reveal the answer?
- Does the ranking correlation hold with more models, including proprietary ones?
2 comments
Rebuttal by Authors (2/2)
Thank you for your detailed questions about our benchmark work. We will address the following four questions.
zURo-Q1
Did the human baseline use full videos or 8 frames?
The same eight frames as the models. Participants viewed the eight uniformly sampled frames used for model evaluation, without access to scene specifications or reference answers, and were scored under the same circular evaluation. The 99.0% vs. 85.9% comparison therefore uses identical visual input. It also shows that the reference answers are recoverable from exactly the input the models receive. We will state this protocol explicitly in revision.
zURo-Q2
Deviation from the specification and re-review of regenerated videos.
Regenerated videos. The 115 videos rejected in the initial Video QC (70 rejected by both reviewers and 45 after adjudicating split cases) were regenerated from the same specifications. The authors re-reviewed each replacement against the original QC criteria: physical implausibility, generation artifacts, and prompt–video drift. This was not another round of independent paired Video QC, and we will state this in App. D.3. The replacements then went through QA QC, where two validators checked every answer against the video (9,800 MCQs in Table 13). All final videos, including replacements, were checked for prompt–video drift. The reported scenario-adherence κ = 0.75 pertains to the initial paired QC.
Deviation on the queried fact. The human baseline provides answer-level consistency evidence. Participants who saw only the eight evaluated frames of each final video, covering all 1,200 videos including the replacements, matched the specification-derived answers on 99.0% of questions under circular evaluation (L1 99.4%, L2 99.3%, L3 97.9%). These results support a high degree of consistency between human answers based on the evaluated frames and the specification-derived references.
However, these results do not tell us how often a video differs from its specification. A participant can make a mistake even when the video is correct, and a participant can still pick the reference answer from the options even when the video is slightly off. We therefore treat the human results as supporting evidence only. We have not directly measured how often the queried fact differs between the video and the specification.
zURo-Q3
Details of the text-oracle information.
We thank the reviewer for asking us to clarify the oracle inputs. In this condition, the frames are replaced by text. The model receives the ground-truth scene graph of the video, which describe objects, attributes (except answers). They are the same for every question on a video and are not selected per question. For example, for a PT–DE question, they dose not describe the direction of the start point from the final viewpoint, which must still be derived.
At the same time, this structured text can substantially simplify the questions, and the condition also changes the input modality from video to text. The increase in L3 accuracy from 40.1% to 90.6% therefore indicates that obtaining usable spatio-temporal information from video is an important bottleneck, but it does not cleanly isolate reasoning ability. We will revise the related statements in revision. We also found that App. E currently describes text oracle inputs as scene graph and scenario, but the input is the scene graph only, so we will correct this wording.
zURo-Q4
Does ranking agreement extend to more models, including proprietary ones?
We thank the reviewer for asking whether the ranking agreement extends to a broader model set. Our current cross-benchmark comparison covers eight open-source MLLMs, yielding Spearman correlations of 0.905 with VSI-Bench and 0.929 with EgoExoBench. These results establish strong ranking agreement within this model set.
To strengthen the validity evidence beyond model rankings, we additionally conducted the matched real–generated study (t19V-Q1). This study compares identical questions and answer options across 50 paired scenarios. Across the eight evaluated models, correctness agreement is 96.2% (95% CI: 95.2–97.3), and the L1-to-L3 accuracy decline is similar on real and generated videos (46.5 and 45.5 pp). At L3, paired correctness agreement is 94.8%. The high correctness agreement therefore shows that, within the tested scenarios, models tend to succeed or fail on the same questions across the two real or generated video sources.
These analyses address complementary aspects of validity. Cross-benchmark correlations assess consistency in relative model performance, while the matched study examines consistency at the question level under controlled scenarios. We will therefore report both forms of evidence in the revision, explicitly scope the ranking results to the evaluated open-source models, and identify extension to proprietary models as an open validation question.
Rebuttal by Authors (1/2)
We thank the reviewer for the constructive feedback and recognition of our controllable benchmark construction and diagnostic analyses. We address the two weaknesses below and the four questions in the following response.
zURo-W1&W2
Video QC agreement. The reported κ = 0.58 was computed as a pooled Cohen’s κ, which was inappropriate for our rotating-pair design. Recomputed as Fleiss’ κ, it remains 0.58 for the overall accept/reject decision, which combines physical plausibility, generation artifacts, and scenario adherence (t19V-Q3). For scenario adherence alone, which is more directly relevant to specification–video alignment, raw agreement is 96.8% and Fleiss’ κ is 0.75. Split cases were adjudicated blind to the identity of the rejecting annotator. These statistics describe agreement during initial screening, rather than the final benchmark’s answer-error rate. We address whether the final videos support the specification-derived answers in Q2.
Beyond the 8-model rank correlation. We conducted a matched real–generated study covering 50 scenarios (t19V-Q1). For each real 8-second video, we generated a counterpart conditioned on its first frame and the same scene specification, and evaluated identical questions at L1, L2, and L3. Across the eight open-source models, paired correctness agreement is 96.2% (95% CI: 95.2–97.3), and the L1-to-L3 accuracy drop is 46.5 pp on real videos versus 45.5 pp on generated videos. This provides complementary evidence that the level-wise decline also occurs in the matched real-video setting, within the scenarios and generation conditions studied. We address the model coverage of the ranking analysis separately in Q4.
One task per taxonomy cell. We agree that one task per cell prevents separating taxonomy-axis effects from task difficulty. The taxonomy organizes scenario construction across spatial scale, perspective, and scene dynamics; it does not establish exhaustive task coverage. We will clarify this scope and report axis-level scores as descriptive summaries of the included tasks.
Our diagnostic comparisons keep task identity fixed: frame shuffling, the text oracle, and answer-option variants change conditions for the same evaluation items. Their effects therefore cannot be explained by task differences across cells. Isolating axis effects would require multiple tasks per cell and matched tasks across axis conditions, which our benchmark does not provide.
Human input. Humans saw the same eight frames as the models, so the human–model comparison uses matched visual input. We clarify the evaluation protocol in Q1.
L3 failures: reasoning or extraction. We agree that the results should not be interpreted as isolating failures of abstract reasoning. Video-based spatio-temporal reasoning requires extracting and organizing visual evidence as well as applying reasoning operations to it. Providing ground-truth facts as text raises L3 accuracy from 40.1% to 90.6%, indicating a substantial contribution from visual extraction, grounding, and representation. Temporal organization also matters: shuffling reduces L3 accuracy by 10.7 pp, compared with 2.4 pp at L1. These results support complementary dependencies, rather than a clean separation of perception and reasoning. Q3 details the oracle inputs and addresses possible answer leakage.
Confidence intervals. We added 95% CIs using 2,000 video-level bootstrap resamples. Across all 15 models, overall accuracy CIs differ approximately 0.8–1.2 pp around each score. Human accuracy is 99.0% (95% CI: 98.7–99.3). On the 120-video ablation set, the L3 shuffle drop exceeds the L1 drop by 8.3 pp (95% CI: 4.1–12.5), and the L3 text-oracle gain is 50.5 pp (95% CI: 45.7–55.3). We will include these estimates and the resampling procedure in the revision.
Novelty relative to VideoHallu. We acknowledge VideoHallu as prior work evaluating MLLMs on generated videos. Its expert-crafted questions test detection of commonsense and physics violations. Our Video QC filters out such defects, and generated videos serve as controlled testbed for spatio-temporal reasoning. Scene graphs and scenarios guide video generation, answers derive from the generation specifications, and human review checks that the videos support them. Our contribution is this specification-driven construction, combined with hierarchical QA and diagnostics requiring ground truth independent of the video. We will clarify this distinction in the Related Work and narrow our novelty claims accordingly.