acceptodds
Withdrawn from ICLR 2027

ACT: Agentic Text–Scene Co-Design and Dual Evolution for Informative Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Informative videos, such as tutorials, advertisements, and public-service videos, combine visual content and on-screen text to deliver specific messages. Generating such videos requires exact text to appear at the requested times and locations without obscuring the main subject. However, current methods often produce incorrect or poorly placed text and fail to coordinate it with the generated scene. To systematically study this problem, we introduce IVBench, a benchmark across five domains that jointly evaluates on-screen text, scene content and layout, and text-scene coordination. To improve text-scene coordination, we propose ACT, an agentic framework for text-scene co-design that plans text and scene together but renders them separately. ACT uses a shared layout contract to reserve space for text and guide subject placement before scene generation. It then adapts text placement to the generated scene, renders the required strings deterministically, and uses verification feedback for targeted repair. To reduce recurring errors and enable persistent system improvement, we further introduce verifier-guided dual evolution, which uses shared execution feedback to update both the harness and the video generator. Experiments on IVBench show that ACT consistently improves text control and text-scene coordination over strong commercial and open-weight baselines, with further gains from dual evolution.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video generation, agentic systems, text-scene coordination, agent evolution, preference optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zJDoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores the problem of informative video generation (e.g., tutorials, advertisements, PSAs), where text must appear at specific timestamps without overlapping the main visual subjects. To address this, the authors introduce IVBench to evaluate informative video generation systems. They also propose an agentic framework that plans spatial layouts for text and subject elements. After video generation, an evaluator agent assesses the output and provides feedback to iteratively refine the layout plan and text prompts. For the rendering stage, the framework uses off-the-shelf video models alongside HTML-rendered text overlays.

Critical strengths

Clear and well-motivated problem formulation. The proposed benchmark and evaluation framework address a crucial gap and provide value to the research community. Extensive experiments and ablation studies effectively support the claims and methodology. Clear and logical pipeline design for the overall framework.

Critical weaknesses

Unclear dataset construction: Section A.1 of the Appendix attempts to describe IVBench's construction, but essential details are omitted. Specifically, it lacks information on video sourcing and licensing compliance, as well as the criteria used to categorize full videos and split them into clips. Furthermore, the paper does not explain how key elements—such as scene descriptions, required strings, display intervals, qualitative layout instructions, and optional references—were extracted. It is also unclear whether text appearing inside objects (e.g., text on book pages) was included as a "required string." Overall, the core dataset creation process is missing from the main manuscript.

Quality control process: The paper does not clarify the human review process for IVBench (e.g., the number of annotators, their domain expertise, and the review protocol). Additionally, details are missing on how model-based evaluators (such as GPT-5.6 and Claude) were prompted and calibrated to review benchmark samples.

Simple baseline: After obtaining the spatial layout from the planner, the pipeline could directly employ a video model that natively accepts bounding boxes or segmentation masks to generate subjects within designated regions. This alternative approach would eliminate the need for a separate verifier. Relevant box/mask-conditioned video generation models include: MagicMotion (Li .et ICCV 2025), VACE ( Jiang .et, ICCV 2025))...

Lack of baselines: The authors do not compare different refiner models. Could a standalone VLM or alternative LLMs be used for this task instead of the proposed agentic framework? General video quality is not checked. (FVD, human evaluation...)

Questions for authors
  • How good Claude is at layout compare to other LLMs model?
  • What is source and authorship, how is dataset curation process?
  • How is video quality of this framework comparing to the base model?
Reviewer_ZwTbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies informative video generation, where specified on-screen text must appear at requested times and locations without occluding the main subject. It contributes IVBench, a 500-sample five-domain benchmark whose seven metrics jointly evaluate text, scene layout, and text-scene coordination, and ACT, a multi-agent framework that plans text and scene together via a shared layout contract, renders text separately with scene-aware compositing, and uses verifier feedback both for targeted instance repair and for dual evolution of the planning/rendering harness (gated strategy edits) and the video generator (direction-verified preference pairs with DPOP). On IVBench, ACT outperforms strong commercial and open-weight baselines on text accuracy, placement, timing, and text-scene separation; ablations and a small human study support each component.

Critical strengths
  1. The paper points out that existing general-purpose benchmarks overlook text-scene coordination, and proposes IVBench to fill this gap. It offers 500 human-written samples across five domains, scored by seven metrics jointly covering on-screen text, scene layout, and text-scene coordination, providing the community with a more suitable evaluation benchmark for informative video generation.

  2. The proposed ACT framework and its verification mechanism are technically sound. The layout contract constrains scene composition before generation, while the structured verifier localizes failures to guide targeted repair.

Critical weaknesses
  1. ACT consumes far more test-time compute than the baselines it is compared against, yet no efficiency comparison and analysis is reported. It remains unclear how much of the gain comes from the method itself rather than from extra compute.

  2. All ablation results (Table 2) come from a single VACE-1.3B backbone on fixed 5-second single-shot clips; without a stronger backbone or longer, multi-shot videos with camera motion, it is unclear whether the framework generalizes beyond this narrow setting.

Questions for authors
  1. What is the wall-clock time and monetary cost per generated video, and how does ACT compare against a compute-matched baseline, for example Best-of-N with the same number of generations and agent calls?

  2. All seven metrics are computed automatically through OCR and VLM-based checks, yet their agreement with human judgment is never measured. Could you report the correlation between each metric and human ratings?

  3. Both repair and evolution depend entirely on verifier feedback, yet the verifier itself is not validated. What is its precision/recall against human labels, and how does the framework degrade when it fails?

Reviewer_kqkCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper is addressing the informative video generation task, where the goal is showing the exact text on the screen at specified times and locations without covering the main subject of the clip. It introduced IVBench with 500 samples across five domains and seven metrics. It also proposed ACT, which is an LLM based agentic pipeline on Wan model that reserves the text space before generation, adds the text with HTML, and runs verification loop. Final dual evolution is refining the harness and fine-tuning the generator. It reports several metrics on IVBench.

Critical strengths
  • It is defining a real failure mode of the current models, i.e. text having a collision with the subject, or unintended timing of the texts.
  • There are many extensive experiments across many baselines.
Critical weaknesses
  • The improvements are due to the construction, where ACT renders the text via deterministic HTML, but baselines mostly generate/draw these in the pixels
  • There is a limited novelty as all the components are from prior works (planning, repairing, HTML rendering).
  • None of the seven metrics is validated/correlated against human evaluation/judgement to see if they are informative.
  • There is no variance analysis across runs or seeds.
  • Even looking at the visual results, gemini-omni looks better (e.g. Figure 13).

I believe the paper is below the standards of this venue. The whole idea is combination of previous works and running the agents.

Questions for authors
  • Can you use native generators combined with your renderer and compositing?
  • What is technically new in this paper?
  • How does the IVBench benchmark align with the human evaluation?
  • What is the source/size of the evolution training?
  • Can you report mean +/- std for all the methods across the seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.