- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces knowledge-intensive video generation, where a model must produce an informative demonstration from a concise information-seeking prompt. It constructs KIVI-Bench with 1,080 prompts across 18 categories and proposes factual precision (FactP), based on extracted and verified textual claims, and helpfulness (HelpS), based on relevance, completeness, and clarity. Nine video generation systems are evaluated on a 54-prompt subset using approximately one-minute videos. A study of 108 pairwise comparisons reports 70.8% and 69.0% agreement with human factuality and helpfulness preferences, respectively, and qualitative analyses identify failures in visual grounding, entity properties, and procedural execution.
Critical strengths
-
The distinction between visually plausible videos and factually useful demonstrations is meaningful. Entity-specific, multi-step requests expose errors that visual-quality metrics can overlook; the complementary factuality and helpfulness dimensions make the task more informative than measuring appearance alone (Sections 3-4).
-
The paper provides useful diagnostic evidence rather than only a leaderboard. Sections 5.2-5.3 separate claim-extraction failures from errors introduced during video generation, and the appendix documents the evaluation prompts and a controlled comparison of interactive versus upfront scripting. These analyses help identify where current pipelines fail.
Critical weaknesses
-
The central factuality metric is insufficiently validated as a measure of the video itself. Verification assumes that the extracted text faithfully describes the video and uses the same LLM's parametric knowledge (Section 3.2, Figure 13). Section 5.2 explicitly shows visually incorrect videos receiving FactP=100% because extraction misstates or abstracts away relevant details. Its prevalence and effect on rankings are not quantified. A same-backbone comparison with direct, content-aware factuality judging is also missing, leaving the benefit of claim decomposition uncertain.
-
Benchmark coverage is limited relative to the scope of the conclusions. All experiments use only 54 of the 1,080 prompts, with three per category (Section 4). Without uncertainty estimates or a broader evaluation, it is difficult to establish that the category-level findings and overall ranking generalize across the proposed benchmark.
-
The human validation does not adequately support the main ranking claims. Each of the 108 comparisons has only one annotator, so inter-annotator reliability is unknown. Appendix A.5 restricts comparisons to three within-tier groups and excludes MiniMax H3 and Seedance 2.5, although these models are central to the leading results. Thus, agreement for the top models and across tiers remains unestablished. Together with the metric-validity concern, this makes me lean toward weak rejection despite the useful task formulation.
Questions for authors
-
Can the authors provide a representative human audit of claim-extraction fidelity and verification accuracy, including missed visual errors, and compare FactP/HelpS with direct factuality/helpfulness judging using the same VLM? The resulting change in human agreement or system ranking would materially affect my assessment.
-
How were the 54 prompts and the 108 model pairs sampled? Can the authors provide uncertainty estimates and overlapping annotations, including comparisons involving MiniMax H3 and Seedance 2.5 and comparisons across model groups?
review by 1nDG
Thank you for the author's response. I will discuss with other reviewers and AC whether to modify the score.