acceptodds
Withdrawn from ICLR 2027

StoryEngine: A State-Grounded Agentic Framework for Video Storytelling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite recent progress in agentic multi-shot video generation, producing coherent and consistent long-form stories remains challenging. Existing agentic pipelines typically rely on textual shot plans or previously generated pixels, yet lack an explicit mechanism for propagating the consequences of story events and maintaining the video world state across shots. As a result, missing visual details may be reconstructed inaccurately, while visual drift may propagate across subsequent shots, undermining both narrative coherence and visual consistency. To address these challenges, we propose StoryEngine, a state-grounded agentic framework for video storytelling. StoryEngine establishes a separation between authoritative semantic plans and unreliable visual observations. Specifically, StoryEngine maintains a structured representation of entity placement and story-relevant states, and propagates event-induced changes to define the intended start and end states of each shot. To visually realize these states, StoryEngine constructs canonical references for recurring entities and environments, and compiles state and visual constraints into executable render plans. Meanwhile, to realize these states correctly, a bounded evaluation-guided repair loop further corrects local state inconsistencies. Together, these mechanisms preserve causal story progression and prevent local visual errors from propagating across shots. To comprehensively evaluate long-form storytelling, we construct a benchmark across diverse scenarios and visual styles, with metrics assessing storytelling quality, narrative coherence, and visual consistency. Experimental results demonstrate that StoryEngine consistently outperforms state-of-the-art methods across all evaluation dimensions, validating its effectiveness for coherent and consistent video storytelling.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Video for Storytelling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_egeKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

StoryEngine proposes a state-grounded agentic framework for long-form video storytelling that explicitly tracks entity and scene states to prevent local generation errors from propagating across shots. It combines story-state propagation, canonical visual grounding with shot-level render planning, and continuity-aware generation with VLM-guided repair to maintain narrative and visual consistency. Experiments show that StoryEngine outperforms ViMax and MovieAgent in narrative realization, cross-shot coherence, and visual consistency, while complex state progression and longer stories remain challenging.

Critical strengths
  1. The paper presents a clear and well-motivated formulation that explicitly separates authoritative story states from fallible generated pixels, effectively addressing error propagation in multi-shot generation.

  2. StoryEngine provides a comprehensive and modular pipeline that integrates state propagation, grounded visual planning, and continuity-aware generation, making the framework easy to interpret and diagnose.

  3. The method shows consistent gains across two different video backbones, suggesting good generalizability beyond a specific video generator.

Critical weaknesses
  1. The paper identifies the lack of explicit world-state propagation as a central limitation of prior work. However, removing the corresponding Story-State Propagation module reduces the overall score by only 1.4 percentage points, indicating a relatively small gain in the ablation study.

  2. This work is more application-oriented. In fact, A²RD[1]’s Multimodal Video Memory already provides a conceptually similar mechanism to the proposed Story-State Propagation: it explicitly maintains textual states, including entity identities, attribute changes, motion, and spatial relations, and updates the memory after each generated segment for use in subsequent generation.

  3. A potential limitation of the plan-first design is that errors in Story-State Propagation may propagate downstream. If a future story-relevant state that induces a visible appearance change is missed during Story-State Propagation, Grounded Render Planning may fail to construct the corresponding state-specific canonical reference. While fixing the semantic state trajectory in advance improves stability, it also makes early planning errors persistent.

  4. StoryEngine already uses Gemini-3.5-Flash as an in-loop evaluator to guide the repair process, while many of the final evaluation metrics are also judged by the same Gemini-3.5-Flash model. As a result, the generated outputs are explicitly optimized to satisfy the preferences and decision boundaries of this judge, and evaluating them again with the same judge may introduce evaluation bias or evaluator overfitting.

  5. The paper introduces new metrics for identity consistency, event completion, and state progression, but why does it not also report standard video-generation metrics such as VBench[2], as done in prior multi-shot video generation works such as HoloCine[3], MultiShotMaster[4], and StoryMem[5]? VBench already provides metrics for subject consistency, human action, scene consistency, and overall consistency, which could complement MIR/ECS/SPS and enable more direct comparison with existing multi-shot generation methods.

[1]: A²RD: Agentic Autoregressive Diffusion for Long Video Consistency

[2]: VBench: Comprehensive Benchmark Suite for Video Generative Models

[3]: HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narratives

[4]: MultiShotMaster: A Controllable Multi-Shot Video Generation Framework

[5]: StoryMem: Multi-shot Long Video Storytelling with Memory

Questions for authors

The paper lacks human evaluation; please see the Weaknesses for the remaining concerns.

Reviewer_BCEZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents StoryEngine, an agentic framework for multi-shot video storytelling. Its central idea is to separate symbolic story state from visual generation: a structured semantic plan is treated as the authoritative state, whereas generated images and videos are treated as potentially fallible observations. The system parses story events, characters, objects, and scene states; constructs canonical visual assets; generates shot plans with start, motion, and end conditions; and performs iterative verification, local repair, and regeneration to improve event coverage, state persistence, and cross-shot continuity. The paper also introduces a 60-story benchmark consisting of three 20-story subsets targeting narrative realization, transitions, and continuity. Experiments are conducted with two video-generation backbones and use metrics for event coverage, state persistence, transition anchors, and character/scene consistency. StoryEngine outperforms the evaluated baselines and ablations on most of the proposed metrics.

Critical strengths
  1. A clear and consequential separation between semantic state and visual evidence. Typed placement relations, explicit event effects, and deterministic state propagation provide a concrete mechanism for maintaining story-relevant facts across shots. Restricting preceding outputs to visual evidence, rather than allowing them to modify story truth, is a sensible approach to preventing local errors from becoming later narrative facts.

  2. A systematic diagnostic evaluation of cross-shot continuity. The benchmark uses frozen annotations that are hidden from the generation systems and separately examines event completion, anchor persistence, irreversible state changes, location, background geometry, identity, and lighting. The experiments cover two video backbones, and the appendix describes the metric protocols in considerable detail. Even if these metrics do not constitute a complete measure of storytelling quality, they provide a potentially useful diagnostic suite for continuity failures.

Critical weaknesses
  1. The task formulation fixes decisions that should arguably be part of storytelling. Each story uses ten shots and two locations, while the input provides the events, location plan, and sometimes transition anchors. The system therefore realizes a largely predefined narrative structure rather than deciding the number of shots, cut positions, visual emphasis, and pacing. These decisions directly affect whether a story is understandable and expressive. The evaluated capability is therefore narrower than the paper’s claim of video storytelling.

  2. The benchmark has limited coverage of storytelling quality. The 60 stories use highly controlled structures designed around event coverage, transitions, and continuity. The benchmark is useful for diagnosing these properties, but it does not adequately evaluate story comprehension, causal clarity, pacing, or visual expressiveness. Related benchmarks such as ViStoryBench [1], LogicTale [2], and StoryBench [3] cover broader narrative dimensions.

  3. The evaluation lacks human validation and efficiency analysis. The paper does not establish whether its automatic metrics reflect human judgments of comprehension, pacing, or overall storytelling quality. Using a Gemini evaluator during both internal verification and final evaluation also creates a risk of evaluator coupling. In addition, StoryEngine may repair or regenerate each shot multiple times, but generation counts, latency, and cost are not reported. Comparisons under matched computation budgets are needed to establish whether the gains come from the proposed framework rather than additional inference.

References

[1] Zhuang et al. ViStoryBench: Comprehensive Benchmark Suite for Story Visualization. CVPR 2026.

[2] Meng et al. LogiStory: A Logic Aware Framework for Multi Image Story Visualization. ICLR 2026.

[3] Bugliarello et al. StoryBench: A Multifaceted Benchmark for Continuous Story Visualization. NeurIPS Datasets and Benchmarks 2023.

Questions for authors
  1. Task formulation: Why must the number of shots be provided by the user? How would StoryEngine perform if it were given only the story and a common duration or compute budget and had to decide the number of shots, event allocation, and cut positions autonomously?

  2. Storytelling quality: Can the authors add human evaluation of story comprehension, causal and action clarity, shot segmentation and pacing, visual expressiveness, and overall preference? How well do PEC, SPS, LAR, GPC, MIR, and LCS correlate with these human judgments?

  3. External validation: Can StoryEngine be evaluated on applicable portions of ViStoryBench, LogicTale, or StoryBench? For example, representative keyframes from the generated shots could be evaluated as a visual-story sequence for plot correspondence, causal logic, action clarity, and expressiveness. If these benchmarks are not applicable, could the authors explain the precise task or modality incompatibilities?

  4. Evaluator independence and computational fairness: How do the authors rule out adaptation to a particular evaluator when a Gemini-based model participates in both generation-time verification and final evaluation? How many candidates, repairs, or regenerations are used on average per story, and does the advantage remain under matched generation counts, compute, or monetary cost?

  5. Related work and novelty: Prior work has already studied LLM based scene planning in VideoDirectorGPT [4] and MovieAgent [7], multi-agent story production in AesopAgent [5], StoryAgent [6], and MM-StoryAgent [8], LogiStory [2]. What technical capability is unique to StoryEngine beyond combining these existing elements? Please clarify whether the main novelty lies in the authoritative state representation, shot phase constraints, canonical assets, continuity gating, or the verification and repair procedure. Direct comparisons or targeted ablations against the most closely related methods would make the novelty easier to assess.

Additional References

[4] Lin et al. VideoDirectorGPT: Consistent Multi Scene Video Generation via LLM Guided Planning. 2023.

[5] Wang et al. AesopAgent: Agent Driven Evolutionary System on Story to Video Production. 2024.

[6] Hu et al. StoryAgent: Customized Storytelling Video Generation via Multi Agent Collaboration. 2024.

[7] Wu et al. Automated Movie Generation via Multi Agent Chain of Thought Planning. 2025.

[8] Xu et al. MM-StoryAgent: Immersive Narrated Storybook Video Generation with a Multi Agent Paradigm across Text, Image and Audio. 2025.

Reviewer_c7gxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes StoryEngine, an agentic framework for multi-shot video storytelling that separates a fixed semantic story plan from potentially unreliable generated images and videos. It represents entity placements and attributes explicitly, propagates event-induced state changes, builds canonical character and environment references, and compiles these into shot-level render contracts. A continuity gate decides whether to reuse, selectively reference, or discard the previous shot's tail frame, while a bounded repair loop attempts to correct violations without changing the semantic plan. Experimental results demonstrate that StoryEngine consistently outperforms state-ofthe-art methods across all evaluation dimensions, validating its effectiveness for coherent and consistent video storytelling.

Critical strengths

The motivation is clear and the framework is complete, turning the idea of separating intended story states from unreliable generated pixels into an end-to-end pipeline. The authors also put considerable effort into the benchmark: annotations are fixed before generation, hidden from all methods, and designed to distinguish event completion, state progression, and visual consistency. Comparisons use the same language, image, and video backends, and the improvements hold across both Veo 3.1 and Wan2.2. The appendix provides detailed state definitions, generation policies, and scoring rules, which makes the system easier to understand and potentially reproduce.

Critical weaknesses

The main concern I have is about the evaluation procedure. StoryEngine uses Gemini-3.5-Flash to guide its repair loop, while the same model judges most of the final evaluation metrics. The paper acknowledges this overlap in Appendix D.10, but does not provide human judgments or results from an independent evaluator. Hidden annotations and method-blind scoring help, but they do not fully address the possibility that the system produces outputs that satisfy the preferences of its own evaluator.

Another issue is that the contribution of explicit state propagation is not fully clear. In Table 2, removing SSP reduces Avg from 0.7690 to 0.7548 and SPS from 0.6681 to 0.6374, while removing grounded render planning causes a substantially larger drop. These results suggest that state propagation helps, but much of the overall gain may come from canonical references and spatial grounding. A more convincing comparison would retain the same references and repair procedure while replacing explicit state propagation with a simpler textual state ledger.

The ablation study is also somewhat incomplete. The w/o SCVC variant removes both tail-based continuity and evaluator-guided repair at once. Since these components address different problems and involve different generation costs, it is difficult to tell whether the improvement comes from selectively carrying visual evidence across shots or simply generating additional candidates. Separately removing the continuity gate and repair loop would make this distinction clearer.

Some evaluation metrics also need stronger validation. LCS rewards differences between lighting conditions without checking whether the requested brighter/darker or warmer/cooler direction is correct. MIR groups faces by appearance rather than matching them to intended characters, which could allow an identity switch to form a new internally consistent cluster. LAR and SPS additionally exclude some unreadable observations. These choices are documented, but their effect on the reported ranking is not examined.

Finally, the practical cost and statistical reliability remain unclear. The paper specifies a repair budget, but does not report actual generation calls, repair frequencies, latency, or monetary cost across methods. The three repeated evaluation runs for PEC, ECS, and GPC also measure scoring variability rather than generation variability. Without generation-level uncertainty estimates, it is difficult to assess the stability of the smaller ablation differences.

Questions for authors

1.Could the authors validate the main ranking using blinded human judgments or an independent VLM that is not used in the repair loop, particularly for event completion and irreversible state progression? 2.Could the authors compare explicit state propagation with a textual state ledger while keeping canonical references, spatial grounding, and repair budgets unchanged? 3.Could the authors separately ablate the continuity gate and evaluator-guided repair, and compare against a baseline with a matched generation or resampling budget? 4.Could the authors report unreadable-observation coverage for LAR and SPS, and test whether identity-switch checks for MIR and directional checks for LCS change the ranking? 5.Could the authors report actual generation costs and paired story-level uncertainty estimates, distinguishing repeated generation from repeated evaluation of the same videos?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.