- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces STORYDYNAMICS, a benchmark and evaluation framework for story visualization. It evaluates generated visual narratives along six dimensions: entity count, character identity, character-object interaction, visual state, state transition, and visual quality. The benchmark contains 93 human-validated English and Chinese stories covering 9 domains, 55 themes, and 4 visual styles. It includes 48 short stories with 5 sentences and 45 long stories with 100 sentences. Each sentence is treated as one narrative step and is required to correspond to one generated frame, resulting in 4,740 frames per evaluated model. The authors construct character references, frame-level state graphs, and character-object interaction annotations from the stories. They evaluate 21 systems spanning general-purpose text-to-image models, trained and training-free story visualization methods, agentic systems, and a video model. The main reported findings are that different model families have complementary strengths, character-object interaction degrades substantially for longer and Chinese stories, and story-specific systems do not consistently outperform general-purpose text-to-image systems. The paper also presents a cinematic sanity check based on a short sequence from a professionally produced film.
Critical strengths
-
The paper evaluates both static content and temporal state changes. Representing character, object, and background states as persistent properties, then separately evaluating whether changes occur at the correct narrative step, is well motivated. This is more informative than evaluating individual images or generic cross-frame similarity alone.
-
The empirical coverage is broad. The paper evaluates 21 systems from several model families on English and Chinese stories, short and long narratives, and multiple visual styles. The appendices expose many prompts and intermediate definitions. This breadth makes the benchmark potentially useful for diagnosing failures in entity interaction, state persistence, and character consistency.
Critical weaknesses
-
The fixed one-sentence-to-one-frame construction excludes storyboard planning. Story visualization requires decisions about shot boundaries, event selection, temporal compression, viewpoint, composition, and pacing. STORYDYNAMICS fixes these choices in advance, so a 100-sentence story always becomes 100 frames regardless of its visual structure. The task therefore resembles sentence-aligned sequential illustration more than end-to-end story visualization. This may favor general-purpose text-to-image models and makes the claimed comparison with story-specific or video systems difficult to interpret.
-
The automated metrics are not validated at the required granularity. Gemma-4-31B-it contributes to both reference construction and prediction scoring, creating shared evaluator bias. The human study covers 192 frames with only three broad state questions and does not directly validate identity, counting, transition timing, semantic matching, or model rankings. For the story-derived interaction configuration actually used, Cohen's kappa with PViC is only 0.287. The identity procedure is also under-specified. Finally, the film experiment derives annotations from the same frames it evaluates and contains only two transition edges, making it a self-consistency check rather than independent validation.
-
The cross-model comparison mixes incompatible protocols. Models receive different narrative context and different forms of character information. Identity is measured against references for some models but through self-consistency for others. These scores are not directly comparable. The overall harmonic mean also gives interaction two components through COI F1 and exact match, and it can collapse because of one near-zero metric. The table is therefore better viewed as a comparison of complete model-interface configurations than a controlled ranking of model capabilities.
-
The length and language findings are confounded and lack uncertainty estimates. The benchmark uses only 5-sentence and 100-sentence stories, while English, Chinese, short, and long results come from different story subsets. Their joint distribution across theme, domain, complexity, and style is not reported. Without matched stories, confidence intervals, or controlled statistical analysis, the observed differences cannot be attributed specifically to length or language.
Questions for authors
-
What is the intended scope of STORYDYNAMICS? Is it designed to evaluate sentence-aligned sequential illustration or full story visualization? Can the authors justify treating every sentence as exactly one shot and quantify how often human storyboarders would choose the same segmentation? A complementary evaluation in which systems can select the number and boundaries of shots would substantially strengthen the holistic claim.
-
Can the authors provide metric-level human validation for entity count, character identity, interaction, visual state, and transition timing? Please report the number of annotators, inter-annotator agreement, sampling design, confidence intervals, and agreement with the automated evaluator at both the frame and model-ranking levels. Can the cinematic validation be repeated with reference annotations created independently of the evaluated frames?
-
What is the exact character identity evaluation procedure, including the evaluator, prompt or representation, comparison pairs, decision rule, and threshold? Can the authors report identity and overall results under a common conditioning and prompting protocol, or separate the rankings by protocol?
-
How are length, language, complexity, style, domain, theme, and interaction density distributed jointly across the 93 stories? Can the authors provide uncertainty estimates and a matched, stratified, or mixed-effects analysis supporting the claims about story length and language?
7 comments
Q3: Character identity evaluation procedure
We clarify that the identity metric and its two comparison settings are defined in Sec. 3.1, while character reference generation is described in Appx. A.4. Sec. 4.1, Appx. B and C.2, and Table 7 clarify generation and evaluation settings. Appx. C.2 specifies Gemma-4-31B-it and the three-vote evaluation configuration. We note that "text-only" refers to the model's generation input, whereas identity evaluation relies on within-story visual comparisons rather than external reference portraits.
We'll include the identity evaluation prompts in Appx. C.2, as follows:
Reference-conditioned models:
Image 1 is the canonical reference portrait of the character [name]. Does Image 2 contain that SAME character? Judge the whole person—hair, outfit, body build, and overall appearance—not only the face: the face may be hidden, turned away, or too small to see, and pose, expression, distance, and lighting may differ. Answer yes or no.
Text-only models:
Image 1 is a reference crop of [name] from earlier in the same picture story. Does Image 2 show the SAME [name] (same identity, design and appearance; pose and lighting may differ)? Answer yes or no.
We note that each response is binary (yes/no), and a match requires at least two of three "yes" votes. No continuous similarity threshold is used. We do not enforce common generation conditioning because models support different inputs. Tables 2 and 11 compare 21 models under their respective settings, rather than isolating conditioning effects. Separating models by conditioning would not eliminate architectural and training differences.
Q4: Story distributions and analysis of length and language effects
We provide additional distributions of 93 stories across length, language, linguistic complexity, visual style, and domain. The benchmark spans 9 domains and 55 themes (1–3 stories per theme). Interaction density denotes ground-truth interactions per frame.
Linguistic Complexity
| Group | Subgroup | Complex | Simple | Total |
|---|---|---|---|---|
| Length | Long | 23 | 22 | 45 |
| Short | 33 | 15 | 48 | |
| Language | English | 29 | 16 | 45 |
| Chinese | 27 | 21 | 48 |
Visual Styles
| Group | Subgroup | Anime | CGI | Claymation | Photorealism | Total |
|---|---|---|---|---|---|---|
| Length | Long | 6 | 15 | 9 | 15 | 45 |
| Short | 14 | 14 | 9 | 11 | 48 | |
| Language | English | 7 | 14 | 8 | 16 | 45 |
| Chinese | 13 | 15 | 10 | 10 | 48 |
Language and Interaction Density
| Group | Subgroup | English | Chinese | Total | Interaction Density |
|---|---|---|---|---|---|
| Length | Long | 21 | 24 | 45 | 0.385 |
| Short | 24 | 24 | 48 | 0.650 | |
| Language | English | 45 | -- | 45 | 0.521 |
| Chinese | -- | 48 | 48 | 0.522 |
Narrative Domains
| Domain | Adventure | Daily Life | Fable | Fantasy | Historical | Mystery | Romance | Sci-Fi | War |
|---|---|---|---|---|---|---|---|---|---|
| Long / Short | 7 / 3 | 1 / 3 | 6 / 6 | 5 / 7 | 5 / 5 | 5 / 4 | 3 / 9 | 6 / 5 | 7 / 6 |
| English / Chinese | 5 / 5 | 1 / 3 | 5 / 7 | 7 / 5 | 4 / 6 | 4 / 5 | 6 / 6 | 4 / 7 | 9 / 4 |
Uncertainty Estimates. We compute 95% CIs using 10,000 story-level bootstrap resamples, retaining all 21 models' results per sampled story.
| Comparison | [95% CI] | Relative Drop [95% CI] | P(Largest Drop Among Metrics) |
|---|---|---|---|
| Long − Short | −0.027 [−0.042, −0.012] | 53.3% [32.8%, 66.0%] | 99.0% |
| Chinese − English | −0.020 [−0.037, −0.004] | 42.0% [10.5%, 61.6%] | 76.2% |
Mixed-Effects Analysis. We fit linear mixed-effects models with random intercepts for model, story, and theme. Fixed effects include length, language, complexity, style, and domain. The second specification additionally controls for interaction density.
| Specification | Long − Short [95% CI] | Chinese − English [95% CI] |
|---|---|---|
| Design-adjusted | −0.030 [−0.049, −0.010] | −0.017 [−0.037, 0.003] |
| + Interaction density | −0.019 [−0.041, 0.003] | −0.019 [−0.040, 0.002] |
Character–object interaction exhibits the largest relative decline for longer stories in 99.0% of bootstrap samples. The length difference remains significant after adjusting for story configurations, but becomes less certain after accounting for interaction density. For language, the observed difference remains negative, although adjusted CIs include zero. We therefore interpret the language gap as an observed association rather than an isolated language effect.
Rebuttal Discussion: Follow-Up on Any Remaining Concerns
Dear Reviewer aKuE,
We would appreciate hearing whether our response has adequately addressed the concerns raised in your review. Your feedback on the clarifications provided would be greatly appreciated.
Thank you again for your time and thoughtful consideration.
Thank you for the thoughtful comments.
Q1: Intended scope of StoryDynamics
StoryDynamics evaluates sentence-aligned story visualizations, with one frame per sentence. We adopt this design for two reasons:
We therefore leave human storyboard segmentation for future exploration.
Q2: Metric-level human validation and cinematic validation
We follow the reviewer's suggestion and extend real-world validation to 12 videos (72 frames). Three annotators are tasked to answer 7 rubric questions per frame, resulting in a total of 1,512 responses. We report Gwet's AC1 [15] for inter-annotator and human–VLM agreement, with 95% bootstrap confidence intervals shown in brackets.
We do not revalidate quality metrics adopted from prior human-aligned methods [12–14]. State-transition scores are computed from state labels rather than separate evaluator judgments; hence no transition AC1 is reported.
Overall, human–VLM agreement reaches AC1 = 0.903, exceeding inter-annotator agreement (0.714), supporting the alignment of our central measurements with human judgments.
References:
[1] Song et al., Character-preserving coherent story visualization, ECCV'20.
[2] Maharana et al., StoryDALL-E: Adapting pretrained text-to-image transformers for story continuation, ECCV'22.
[3] Li & Lukasiewicz, Learning to model multimodal semantic alignment for story visualization, EMNLP'22.
[4] Ahn et al., Story visualization by online text augmentation with context memory, ICCV'23.
[5] Luo et al., Object isolated attention for consistent story visualization, ICME'25.
[6] Park et al., Infinite-Story: A training-free consistent text-to-image generation, AAAI'26.
[7] Dong & Bargal, Sidecar: Training-Free Semantic Reuse for Character-Consistent Free-form Visual Storytelling, arXiv'26.
[8] Michelmann et al., Large language models can segment narrative events similarly to humans, Behavior Research Methods'25.
[9] Hodgetts et al., Individual differences in experiential diversity shape event segmentation granularity, iScience'25.
[10] Güler & Günseli, The mosaic of experience: How individual differences in attention and working memory shape event segmentation, Cognitive Research: Principles and Implications'26.
[11] Baror et al., The role of context in continuity and segmentation, Nature Human Behaviour'26.
[12] Wang et al., Unified Reward Model for Multimodal Understanding and Generation, arXiv'25.
[13] Schuhmann et al., LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models, NeurIPS'22.
[14] Schuhmann, LAION-Aesthetics, LAION'22.
[15] Gwet, Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement, British Journal of Mathematical and Statistical Psychology'08.