- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper explores the problem of informative video generation (e.g., tutorials, advertisements, PSAs), where text must appear at specific timestamps without overlapping the main visual subjects. To address this, the authors introduce IVBench to evaluate informative video generation systems. They also propose an agentic framework that plans spatial layouts for text and subject elements. After video generation, an evaluator agent assesses the output and provides feedback to iteratively refine the layout plan and text prompts. For the rendering stage, the framework uses off-the-shelf video models alongside HTML-rendered text overlays.
Critical strengths
Clear and well-motivated problem formulation. The proposed benchmark and evaluation framework address a crucial gap and provide value to the research community. Extensive experiments and ablation studies effectively support the claims and methodology. Clear and logical pipeline design for the overall framework.
Critical weaknesses
Unclear dataset construction: Section A.1 of the Appendix attempts to describe IVBench's construction, but essential details are omitted. Specifically, it lacks information on video sourcing and licensing compliance, as well as the criteria used to categorize full videos and split them into clips. Furthermore, the paper does not explain how key elements—such as scene descriptions, required strings, display intervals, qualitative layout instructions, and optional references—were extracted. It is also unclear whether text appearing inside objects (e.g., text on book pages) was included as a "required string." Overall, the core dataset creation process is missing from the main manuscript.
Quality control process: The paper does not clarify the human review process for IVBench (e.g., the number of annotators, their domain expertise, and the review protocol). Additionally, details are missing on how model-based evaluators (such as GPT-5.6 and Claude) were prompted and calibrated to review benchmark samples.
Simple baseline: After obtaining the spatial layout from the planner, the pipeline could directly employ a video model that natively accepts bounding boxes or segmentation masks to generate subjects within designated regions. This alternative approach would eliminate the need for a separate verifier. Relevant box/mask-conditioned video generation models include: MagicMotion (Li .et ICCV 2025), VACE ( Jiang .et, ICCV 2025))...
Lack of baselines: The authors do not compare different refiner models. Could a standalone VLM or alternative LLMs be used for this task instead of the proposed agentic framework? General video quality is not checked. (FVD, human evaluation...)
Questions for authors
- How good Claude is at layout compare to other LLMs model?
- What is source and authorship, how is dataset curation process?
- How is video quality of this framework comparing to the base model?