acceptodds
Under review as a conference paper at ICLR 2027

Knowledge-Intensive Video Generation

Abstract

Text-to-video generation has advanced rapidly in visual quality, but existing task settings largely assume that the prompt already specifies what the video should depict. This assumption does not hold in information-seeking scenarios, where users typically specify what they want to learn rather than the detailed visual content of the answer. We introduce Knowledge-Intensive Video Generation (KIVI), where models must determine and generate appropriate visual content from concise information-seeking prompts that request explanations, procedures, or demonstrations. Many such requests naturally require long videos that organize multiple factual or procedural details over time. To study this capability, we construct KIVI-Bench, a benchmark of 1,080 prompts targeting long-form video generation, and propose automatic metrics for factuality and helpfulness. Human evaluation shows that our metrics achieve substantially stronger agreement with human judgments than existing visual-quality-oriented alternatives. Experiments on nine state-of-the-art video generation models reveal substantial variation in KIVI performance, while even the strongest models exhibit frequent failures in entity-specific visual properties, procedural operations, and clear information presentation. These results highlight the need to move beyond visually plausible rendering toward video generation systems that can reliably communicate correct and useful information.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
text-to-video generation, factuality, video generation, knowledge-intensive generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wirPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces knowledge-intensive video generation, where a model must produce an informative demonstration from a concise information-seeking prompt. It constructs KIVI-Bench with 1,080 prompts across 18 categories and proposes factual precision (FactP), based on extracted and verified textual claims, and helpfulness (HelpS), based on relevance, completeness, and clarity. Nine video generation systems are evaluated on a 54-prompt subset using approximately one-minute videos. A study of 108 pairwise comparisons reports 70.8% and 69.0% agreement with human factuality and helpfulness preferences, respectively, and qualitative analyses identify failures in visual grounding, entity properties, and procedural execution.

Critical strengths
  1. The distinction between visually plausible videos and factually useful demonstrations is meaningful. Entity-specific, multi-step requests expose errors that visual-quality metrics can overlook; the complementary factuality and helpfulness dimensions make the task more informative than measuring appearance alone (Sections 3-4).

  2. The paper provides useful diagnostic evidence rather than only a leaderboard. Sections 5.2-5.3 separate claim-extraction failures from errors introduced during video generation, and the appendix documents the evaluation prompts and a controlled comparison of interactive versus upfront scripting. These analyses help identify where current pipelines fail.

Critical weaknesses
  1. The central factuality metric is insufficiently validated as a measure of the video itself. Verification assumes that the extracted text faithfully describes the video and uses the same LLM's parametric knowledge (Section 3.2, Figure 13). Section 5.2 explicitly shows visually incorrect videos receiving FactP=100% because extraction misstates or abstracts away relevant details. Its prevalence and effect on rankings are not quantified. A same-backbone comparison with direct, content-aware factuality judging is also missing, leaving the benefit of claim decomposition uncertain.

  2. Benchmark coverage is limited relative to the scope of the conclusions. All experiments use only 54 of the 1,080 prompts, with three per category (Section 4). Without uncertainty estimates or a broader evaluation, it is difficult to establish that the category-level findings and overall ranking generalize across the proposed benchmark.

  3. The human validation does not adequately support the main ranking claims. Each of the 108 comparisons has only one annotator, so inter-annotator reliability is unknown. Appendix A.5 restricts comparisons to three within-tier groups and excludes MiniMax H3 and Seedance 2.5, although these models are central to the leading results. Thus, agreement for the top models and across tiers remains unestablished. Together with the metric-validity concern, this makes me lean toward weak rejection despite the useful task formulation.

Questions for authors
  1. Can the authors provide a representative human audit of claim-extraction fidelity and verification accuracy, including missed visual errors, and compare FactP/HelpS with direct factuality/helpfulness judging using the same VLM? The resulting change in human agreement or system ranking would materially affect my assessment.

  2. How were the 54 prompts and the 108 model pairs sampled? Can the authors provide uncertainty estimates and overlapping annotations, including comparisons involving MiniMax H3 and Seedance 2.5 and comparisons across model groups?

Reviewer_fWi2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces knowledge-intensive video generation (KIVI), where models must produce factually accurate and useful videos from concise information-seeking prompts. The authors construct KIVI-Bench, containing 1,080 prompts across 18 categories, and propose two automatic metrics: Factual Precision (FactP) and Helpfulness Score (HelpS). They evaluate nine video generation models on a 54-prompt subset and validate the metrics through 108 human pairwise comparisons. The results suggest that conventional visual-quality metrics do not adequately capture factual correctness or instructional usefulness. The task is meaningful, but the evidence supporting the reliability of the proposed metrics and the generalizability of the experimental findings remains insufficient.

Critical strengths
  1. A practically relevant task formulation. Users often know what they want to learn without knowing the visual content or procedural details needed to explain it. Evaluating whether generated videos can accurately address such requests is valuable for instructional and information-seeking applications.

  2. A useful distinction between factuality and helpfulness. A video can contain correct statements while omitting essential steps or presenting an unfollowable procedure. Evaluating both dimensions is therefore better aligned with the task than relying exclusively on visual quality.

  3. Informative qualitative analysis. The paper identifies entity-specific appearance errors, component misplacement, incorrect procedures, and failures occurring despite correct outlines. It also acknowledges limitations in automatic claim extraction, providing useful examples of how evaluation can overlook visual errors.

Critical weaknesses
  1. Factual verification lacks independent, traceable evidence. In the main evaluation, Gemini extracts textual claims from generated videos and subsequently verifies them using its parametric knowledge. Firstly, the verification stage does not use retrieved authoritative documentation, validated reference videos, or human-curated factual annotations. Secondly, it explicitly assumes that the extracted claims faithfully describe the video.

    This leaves two essential assumptions insufficiently validated: that claim extraction is faithful and comprehensive, and that the verifier possesses accurate knowledge of the specific task. These assumptions are particularly consequential for product-specific features, software versions, and specialized procedures. The paper does not establish claim-level verification accuracy against an independent human-validated standard. Asking the verifier to provide references is not equivalent to retrieving and checking supporting evidence.

  2. Agreement with human preferences does not establish adequate error-detection capability.

    Text-only verification achieves higher agreement with human preferences than the tested multimodal alternatives. This does not demonstrate that it more accurately detects factual and visual errors. The explanation that visual evidence introduces distracting context remains a hypothesis. Beyond pairwise preference agreement, the metrics should be evaluated for their detection rates and false-positive rates across error types, particularly serious procedural errors and mismatches between textual claims and actual visual content. The current validation does not sufficiently establish these capabilities.

  3. The experimental sample is too small to support broad conclusions without uncertainty analysis. Although KIVI-Bench contains 1,080 prompts, all experiments use only 54, with three prompts per category. This supports exploratory observations but provides limited evidence for category-level conclusions and stable model rankings.

    The paper does not report confidence intervals, repeated-subset evaluations, repeated-generation variability, or ranking stability. Multiple claims extracted from the same video cannot substitute for independent tasks because they share the same prompt and generation process. Larger-scale evaluation and appropriate uncertainty analysis is needed.

  4. It is unclear whether the models receive identical planning outputs.

    All systems use Gemini 3.1 Pro with temperature zero for planning and script generation, but the paper does not specify whether an outline is generated once per prompt and reused across all video models, or generated independently for each model. Using the same planner and decoding settings does not establish that the supplied procedural knowledge and details are identical.

    Moreover, subsequent segment scripts depend on each model’s previously generated video. Different outputs can therefore lead the planner to provide different instructions and knowledge details. The resulting scores reflect the interaction between the video generator and the adaptive planner, complicating attribution of performance differences to the video model alone.

    The authors should clarify their generation protocol and compare fixed-outline/script evaluation with adaptive scripting. This would help separate video-generation capability from planner variation and adaptation.

  5. Potential overlap with training data. Some evaluated models may have encountered the reference videos or closely related tutorials during training, making it difficult to distinguish generalization from memorization. To mitigate this risk, the authors could create controlled variants of the collected tasks, for example, by changing the device model, software version, or task constraints, rather than evaluating only the original tasks found online. These modifications should require meaningful changes to the demonstrated procedure, with the correct solution independently verified. Such variants would reduce reliance on memorized tutorials and better test whether models can adapt their knowledge to new settings, although they would not eliminate contamination risk entirely.

Questions for authors

please see weaknesses

Reviewer_1nDGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper formulates knowledge-intensive video generation, where models must generate factually accurate and instructionally useful videos from short information-seeking prompts rather than fully specified scene descriptions. The authors construct KIVI-Bench with 1,080 prompts across 18 WikiHow Video topic categories, using GPT-generated candidates and two-stage human quality control. They propose two LLM-based metrics: Factual Precision, which extracts video claims and verifies their correctness, and Helpfulness Score, which rates Relevance, Completeness, and Clarity. Human evaluation on 108 pairwise comparisons reports 70.8% and 69.0% agreement with human judgments, and the authors benchmark nine video generation models on a 54-prompt subset.

Critical strengths
  • The task setting is timely and important: it shifts text-to-video evaluation from visual plausibility toward factual accuracy and instructional usefulness.
  • The benchmark construction has meaningful quality control, including unique entities, verifiable answers, non-obvious knowledge, and human filtering of ambiguous or unsupported prompts.
  • The proposed FactP and HelpS metrics target complementary aspects, and human validation suggests they align better with human judgments than several VBench-Long visual-quality dimensions.
  • The error analysis identifies recurring failures such as entity misrepresentation, incorrect procedure, and component misplacement, which are useful for future work.
Critical weaknesses
  • Human validation is limited to 108 pairwise comparisons with one annotator per pair and no reported inter-annotator agreement, confidence intervals, or significance tests. 
  • FactP relies on text-only claim extraction and verification, which the authors acknowledge can miss or abstract away important visual details and may overestimate factuality. 
  • The evaluation pipeline depends heavily on a single LLM, Gemini 3.1 Pro, for planning, claim extraction, verification, and helpfulness scoring; robustness across evaluators is not established.
Questions for authors
  • Can the authors report results on the full 1,080-prompt benchmark, or at least justify the 54-prompt subset with a stratified sampling and power analysis? 

  • Can the human evaluation be expanded with more annotators, inter-annotator agreement, confidence intervals, and significance tests? 

  • How often does text-only claim extraction miss visually grounded errors, and can the authors quantify the resulting overestimation of FactP? 

  • Would FactP/HelpS and the model rankings remain stable if a different LLM were used for claim extraction and verification?

  • The authors acknowledge that text-only claim verification may miss visually grounded errors. Can the authors add an experiment that directly evaluates and compares the performance of their text-only FactP against a fully multimodal verification setup (e.g., feeding the video frames directly to a Vision-Language Model for claim checking) on a subset of the benchmark?

1 comment
Reviewer_1nDGOpenReview ↗

review by 1nDG

Thank you for the author's response. I will discuss with other reviewers and AC whether to modify the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.