acceptodds
Withdrawn from ICLR 2027

StoryDynamics: Holistic Evaluation of Story Visualization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A coherent story visualization requires more than generating individually plausible images. Correct entities should appear, recurring characters should remain recognizable, their interactions should reflect the narrative, visual states should be faithfully depicted, those states should change at the appropriate point in the story, and the resulting frames should remain visually faithful to the text. Existing benchmarks capture subsets of these properties, but do not provide a unified evaluation of them jointly. We introduce StoryDynamics, an evaluation framework that assesses story visualization across six complementary dimensions: entity count, character identity, character-object interaction, visual state, state transition, and visual quality. To support this evaluation, we construct a benchmark of 93 human-validated English and Chinese stories spanning diverse narrative lengths, domains, themes, and visual styles. Each story is paired with the reference information needed to evaluate both frame-level properties and how observable states evolve throughout the narrative. Using StoryDynamics, we evaluate 21 general-purpose and story-specific generation models. Our benchmark evaluation shows that strong performance on individual dimensions does not necessarily translate to coherent story visualization, that character-object interaction is the most vulnerable capability across story length and language, and that story-specific models do not consistently achieve holistic gains over general-purpose text-to-image models. Our results highlight the importance of jointly evaluating both static and temporal properties when assessing story visualization models.

Primary area:
datasets and benchmarks
Keywords:
Story Visualization, Evaluation, Benchmark, State Transitions, Holistic Story Visualization Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aKuEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces STORYDYNAMICS, a benchmark and evaluation framework for story visualization. It evaluates generated visual narratives along six dimensions: entity count, character identity, character-object interaction, visual state, state transition, and visual quality. The benchmark contains 93 human-validated English and Chinese stories covering 9 domains, 55 themes, and 4 visual styles. It includes 48 short stories with 5 sentences and 45 long stories with 100 sentences. Each sentence is treated as one narrative step and is required to correspond to one generated frame, resulting in 4,740 frames per evaluated model. The authors construct character references, frame-level state graphs, and character-object interaction annotations from the stories. They evaluate 21 systems spanning general-purpose text-to-image models, trained and training-free story visualization methods, agentic systems, and a video model. The main reported findings are that different model families have complementary strengths, character-object interaction degrades substantially for longer and Chinese stories, and story-specific systems do not consistently outperform general-purpose text-to-image systems. The paper also presents a cinematic sanity check based on a short sequence from a professionally produced film.

Critical strengths
  1. The paper evaluates both static content and temporal state changes. Representing character, object, and background states as persistent properties, then separately evaluating whether changes occur at the correct narrative step, is well motivated. This is more informative than evaluating individual images or generic cross-frame similarity alone.

  2. The empirical coverage is broad. The paper evaluates 21 systems from several model families on English and Chinese stories, short and long narratives, and multiple visual styles. The appendices expose many prompts and intermediate definitions. This breadth makes the benchmark potentially useful for diagnosing failures in entity interaction, state persistence, and character consistency.

Critical weaknesses
  1. The fixed one-sentence-to-one-frame construction excludes storyboard planning. Story visualization requires decisions about shot boundaries, event selection, temporal compression, viewpoint, composition, and pacing. STORYDYNAMICS fixes these choices in advance, so a 100-sentence story always becomes 100 frames regardless of its visual structure. The task therefore resembles sentence-aligned sequential illustration more than end-to-end story visualization. This may favor general-purpose text-to-image models and makes the claimed comparison with story-specific or video systems difficult to interpret.

  2. The automated metrics are not validated at the required granularity. Gemma-4-31B-it contributes to both reference construction and prediction scoring, creating shared evaluator bias. The human study covers 192 frames with only three broad state questions and does not directly validate identity, counting, transition timing, semantic matching, or model rankings. For the story-derived interaction configuration actually used, Cohen's kappa with PViC is only 0.287. The identity procedure is also under-specified. Finally, the film experiment derives annotations from the same frames it evaluates and contains only two transition edges, making it a self-consistency check rather than independent validation.

  3. The cross-model comparison mixes incompatible protocols. Models receive different narrative context and different forms of character information. Identity is measured against references for some models but through self-consistency for others. These scores are not directly comparable. The overall harmonic mean also gives interaction two components through COI F1 and exact match, and it can collapse because of one near-zero metric. The table is therefore better viewed as a comparison of complete model-interface configurations than a controlled ranking of model capabilities.

  4. The length and language findings are confounded and lack uncertainty estimates. The benchmark uses only 5-sentence and 100-sentence stories, while English, Chinese, short, and long results come from different story subsets. Their joint distribution across theme, domain, complexity, and style is not reported. Without matched stories, confidence intervals, or controlled statistical analysis, the observed differences cannot be attributed specifically to length or language.

Questions for authors
  1. What is the intended scope of STORYDYNAMICS? Is it designed to evaluate sentence-aligned sequential illustration or full story visualization? Can the authors justify treating every sentence as exactly one shot and quantify how often human storyboarders would choose the same segmentation? A complementary evaluation in which systems can select the number and boundaries of shots would substantially strengthen the holistic claim.

  2. Can the authors provide metric-level human validation for entity count, character identity, interaction, visual state, and transition timing? Please report the number of annotators, inter-annotator agreement, sampling design, confidence intervals, and agreement with the automated evaluator at both the frame and model-ranking levels. Can the cinematic validation be repeated with reference annotations created independently of the evaluated frames?

  3. What is the exact character identity evaluation procedure, including the evaluator, prompt or representation, comparison pairs, decision rule, and threshold? Can the authors report identity and overall results under a common conditioning and prompting protocol, or separate the rankings by protocol?

  4. How are length, language, complexity, style, domain, theme, and interaction density distributed jointly across the 93 stories? Can the authors provide uncertainty estimates and a matched, stratified, or mixed-effects analysis supporting the claims about story length and language?

7 comments

Thank you for the thoughtful comments.

Q1: Intended scope of StoryDynamics

StoryDynamics evaluates sentence-aligned story visualizations, with one frame per sentence. We adopt this design for two reasons:

  1. Consistency with prior works. We follow existing story visualization protocols [1–7], enabling standardized comparisons and evaluation of our proposed metrics.
  2. Reducing segmentation subjectivity. Fixed sentence-level segmentation mitigates subjectivity in narrative boundaries, which can vary with experience, attention, and context [8–11].

We therefore leave human storyboard segmentation for future exploration.

Q2: Metric-level human validation and cinematic validation

We follow the reviewer's suggestion and extend real-world validation to 12 videos (72 frames). Three annotators are tasked to answer 7 rubric questions per frame, resulting in a total of 1,512 responses. We report Gwet's AC1 [15] for inter-annotator and human–VLM agreement, with 95% bootstrap confidence intervals shown in brackets.

MetricsCntchar\mathrm{Cnt}_{char}Cntobj\mathrm{Cnt}_{obj}ID\mathrm{ID}F1COI\mathrm{F1}_{COI}EMCOI\mathrm{EM}_{COI}StateF1\mathrm{StateF1}TransF1\mathrm{TransF1}Overall
Human Score0.8520.9000.9220.9130.7670.9800.8050.872
VLM Score0.8470.9310.9270.8610.7750.8200.6840.827
Inter-annotator AC10.609 [.29,.84]0.726 [.41,.93]0.736 [.48,.91]0.751 [.54,.91]0.751 [.54,.91]0.723 [.59,.81]--0.714 [.61,.80]
Human–VLM AC10.844 [.71,.95]0.876 [.75,.98]0.904 [.81,.97]0.846 [.71,.95]0.807 [.67,.92]0.968 [.94,.99]--0.903 [.87,.93]

We do not revalidate quality metrics adopted from prior human-aligned methods [12–14]. State-transition scores are computed from state labels rather than separate evaluator judgments; hence no transition AC1 is reported.

Overall, human–VLM agreement reaches AC1 = 0.903, exceeding inter-annotator agreement (0.714), supporting the alignment of our central measurements with human judgments.

References:

[1] Song et al., Character-preserving coherent story visualization, ECCV'20.

[2] Maharana et al., StoryDALL-E: Adapting pretrained text-to-image transformers for story continuation, ECCV'22.

[3] Li & Lukasiewicz, Learning to model multimodal semantic alignment for story visualization, EMNLP'22.

[4] Ahn et al., Story visualization by online text augmentation with context memory, ICCV'23.

[5] Luo et al., Object isolated attention for consistent story visualization, ICME'25.

[6] Park et al., Infinite-Story: A training-free consistent text-to-image generation, AAAI'26.

[7] Dong & Bargal, Sidecar: Training-Free Semantic Reuse for Character-Consistent Free-form Visual Storytelling, arXiv'26.

[8] Michelmann et al., Large language models can segment narrative events similarly to humans, Behavior Research Methods'25.

[9] Hodgetts et al., Individual differences in experiential diversity shape event segmentation granularity, iScience'25.

[10] Güler & Günseli, The mosaic of experience: How individual differences in attention and working memory shape event segmentation, Cognitive Research: Principles and Implications'26.

[11] Baror et al., The role of context in continuity and segmentation, Nature Human Behaviour'26.

[12] Wang et al., Unified Reward Model for Multimodal Understanding and Generation, arXiv'25.

[13] Schuhmann et al., LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models, NeurIPS'22.

[14] Schuhmann, LAION-Aesthetics, LAION'22.

[15] Gwet, Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement, British Journal of Mathematical and Statistical Psychology'08.

Q3: Character identity evaluation procedure

We clarify that the identity metric and its two comparison settings are defined in Sec. 3.1, while character reference generation is described in Appx. A.4. Sec. 4.1, Appx. B and C.2, and Table 7 clarify generation and evaluation settings. Appx. C.2 specifies Gemma-4-31B-it and the three-vote evaluation configuration. We note that "text-only" refers to the model's generation input, whereas identity evaluation relies on within-story visual comparisons rather than external reference portraits.

We'll include the identity evaluation prompts in Appx. C.2, as follows:

Reference-conditioned models:

Image 1 is the canonical reference portrait of the character [name]. Does Image 2 contain that SAME character? Judge the whole person—hair, outfit, body build, and overall appearance—not only the face: the face may be hidden, turned away, or too small to see, and pose, expression, distance, and lighting may differ. Answer yes or no.

Text-only models:

Image 1 is a reference crop of [name] from earlier in the same picture story. Does Image 2 show the SAME [name] (same identity, design and appearance; pose and lighting may differ)? Answer yes or no.

We note that each response is binary (yes/no), and a match requires at least two of three "yes" votes. No continuous similarity threshold is used. We do not enforce common generation conditioning because models support different inputs. Tables 2 and 11 compare 21 models under their respective settings, rather than isolating conditioning effects. Separating models by conditioning would not eliminate architectural and training differences.

Q4: Story distributions and analysis of length and language effects

We provide additional distributions of 93 stories across length, language, linguistic complexity, visual style, and domain. The benchmark spans 9 domains and 55 themes (1–3 stories per theme). Interaction density denotes ground-truth interactions per frame.

Linguistic Complexity

GroupSubgroupComplexSimpleTotal
LengthLong232245
Short331548
LanguageEnglish291645
Chinese272148

Visual Styles

GroupSubgroupAnimeCGIClaymationPhotorealismTotal
LengthLong61591545
Short141491148
LanguageEnglish71481645
Chinese1315101048

Language and Interaction Density

GroupSubgroupEnglishChineseTotalInteraction Density
LengthLong2124450.385
Short2424480.650
LanguageEnglish45--450.521
Chinese--48480.522

Narrative Domains

DomainAdventureDaily LifeFableFantasyHistoricalMysteryRomanceSci-FiWar
Long / Short7 / 31 / 36 / 65 / 75 / 55 / 43 / 96 / 57 / 6
English / Chinese5 / 51 / 35 / 77 / 54 / 64 / 56 / 64 / 79 / 4

Uncertainty Estimates. We compute 95% CIs using 10,000 story-level bootstrap resamples, retaining all 21 models' results per sampled story.

ComparisonΔF1COI\Delta\mathrm{F1}_{COI} [95% CI]Relative Drop [95% CI]P(Largest Drop Among Metrics)
Long − Short−0.027 [−0.042, −0.012]53.3% [32.8%, 66.0%]99.0%
Chinese − English−0.020 [−0.037, −0.004]42.0% [10.5%, 61.6%]76.2%

Mixed-Effects Analysis. We fit linear mixed-effects models with random intercepts for model, story, and theme. Fixed effects include length, language, complexity, style, and domain. The second specification additionally controls for interaction density.

SpecificationLong − Short [95% CI]Chinese − English [95% CI]
Design-adjusted−0.030 [−0.049, −0.010]−0.017 [−0.037, 0.003]
+ Interaction density−0.019 [−0.041, 0.003]−0.019 [−0.040, 0.002]

Character–object interaction exhibits the largest relative decline for longer stories in 99.0% of bootstrap samples. The length difference remains significant after adjusting for story configurations, but becomes less certain after accounting for interaction density. For language, the observed difference remains negative, although adjusted CIs include zero. We therefore interpret the language gap as an observed association rather than an isolated language effect.

AuthorsOpenReview ↗

Rebuttal Discussion: Follow-Up on Any Remaining Concerns

Dear Reviewer aKuE,

We would appreciate hearing whether our response has adequately addressed the concerns raised in your review. Your feedback on the clarifications provided would be greatly appreciated.

Thank you again for your time and thoughtful consideration.

Reviewer_Vk5xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces StoryDynamics, a benchmark and evaluation framework for assessing story visualization beyond individual-image quality and character consistency. It evaluates six dimensions: entity count, character identity, character–object interaction, visual state, state transition, and visual quality. The benchmark comprises 93 human-validated English and Chinese stories spanning different lengths, linguistic complexities, domains, and visual styles, totaling 4,740 frames per evaluated model. A VLM-based pipeline compares generated images against structured narrative annotations to assess frame-level correctness and state evolution across the story. Experiments evaluate 21 general-purpose and story-specific models, supplemented by evaluator validation and a professional film example. The results highlight weaknesses in character–object interactions and show that story-specific models do not consistently outperform general-purpose models across the evaluated dimensions.

Critical strengths
  1. Well-motivated evaluation scope. The framework jointly examines entity presence, identity, interactions, states, transitions, and visual quality, addressing important aspects of narrative coherence that individual-image metrics miss.

  2. Explicit modeling of narrative state evolution. Structured state annotations distinguish whether a visual state is correct from whether it changes at the appropriate story step, providing a useful diagnostic of temporal narrative fidelity.

  3. Broad empirical coverage. Evaluation spans 21 models and both short and long English and Chinese stories across diverse domains and styles, revealing capability gaps that aggregate image-quality scores can obscure.

Critical weaknesses
  1. Reference-based identification introduces an untested confound. Although text-only models receive character descriptions, entity and interaction extraction still identifies characters using reference portraits (Appendix C.2). The paper does not establish whether deviations from these portraits cause otherwise consistent characters to be excluded, potentially conflating appearance matching with narrative correctness.

  2. Evaluator validation has limited coverage. Human validation assesses aggregated visual-state judgments across six methods, without directly validating identity or state-transition accuracy. The PViC comparison provides additional interaction validation, but agreement under the benchmark’s story-derived vocabulary is limited (κ = 0.287). These checks do not fully establish reliability across the proposed dimensions.

Questions for authors

Given that entity and interaction extraction uses reference portraits to identify characters, can the authors assess whether this penalizes text-only models despite their access to character descriptions, and whether an alternative identification protocol changes the rankings?

4 comments

Thanks for the insightful comments.

Q1: Limited evaluator validation and agreement under story-derived vocabulary

Our study includes 2,304 human judgments, comparable to VPEval (1,000), TIFA (1,600), and HumanScore (~1,200) [1–3]. We additionally conduct frame-level validation with 3 human annotators on 72 frames from 12 real-world videos, using the same rubric as our VLM evaluator.

MetricsCntchar↑\mathrm{Cnt}_{char}\uparrowCntobj↑\mathrm{Cnt}_{obj}\uparrowID↑\mathrm{ID}\uparrowF1COI↑\mathrm{F1}_{COI}\uparrowEMCOI↑\mathrm{EM}_{COI}\uparrowStateF1↑\mathrm{StateF1}\uparrowTransF1↑\mathrm{TransF1}\uparrowOverall↑\mathrm{Overall}\uparrow
Human Score0.8520.9000.9220.9130.7670.9800.8050.872
VLM Score0.8470.9310.9270.8610.7750.8200.6840.827
Gwet's AC10.8440.8760.9040.8460.8070.968-0.903

We observe high human-VLM agreement across the evaluated dimensions, with an overall AC1 of 0.903.

Regarding PViC, the story-derived vocabulary is substantially larger than PViC's:

VocabularyVerbsObjectsVerb–Object Pairs
Story-derived4317881,298
PViC11780600
Covered by PViC51.5%26.1%10.7%

For frames where only PViC detects an interaction, our VLM evaluators can identify the interaction, but it is lost when mapped to PViC labels.

Furthermore, Cohen's κ\kappa accounts for chance agreement. Since 80-85% of frames contain no matching interaction, the expected chance agreement is high, resulting in a lower κ\kappa despite 79.4% observed agreement.

Object / Verb VocabularyAgreementChance Agreementκ\kappa
Story-derived / Story-derived0.7940.7110.287
PViC / PViC0.9800.7990.902

Under aligned PViC vocabulary constraints, agreement increases to 98.0% (κ=0.902\kappa=0.902). This demonstrates strong agreement under matched vocabulary, while the broader story-derived setting remains more challenging.

Q2: Impact of reference-based character identification on text-only models

We follow the reviewer's suggestion and re-evaluate models using text descriptions instead of reference portraits for character identification. Ref and Text denote the two evaluator settings.

CategoryModelOverall (Ref)Overall (Text)Rank (Ref→Text)
Text-to-ImageFlux.10.0500.0949→6
SDXL0.0350.07710→8
Stable Diffusion 3.50.0590.0887→7
CharaConsist0.0550.0758→9
Story VisualizationStoryDiffusion0.0320.02911→15
Story Iter0.0070.05714→12
ConsiStory0.0010.00718→18
One-prompt-one-story0.0020.00417→20
Anim-Director0.0120.04413→13
MMStoryAgent0.0000.02221→16
MovieAgent0.0240.06712→10

Overall scores generally increase under text-only descriptions, although some rankings change. These results show that the identification protocol influences absolute scores and individual rankings, while the broader evaluation findings remain informative.

References:

[1] Cho et al., 2023b, Visual programming for step-by-step text-to-image generation and evaluation, NeurIPS'23.

[2] Hu et al., 2023, TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering, ICCV'23.

[3] Xiang et al., 2026, HumanScore: Benchmarking human motions in generated videos, arXiv'26.

AuthorsOpenReview ↗

Rebuttal Discussion: Follow-Up on Any Remaining Concerns

Dear Reviewer Vk5x,

We would appreciate any further feedback you may have on our response, particularly regarding whether our clarifications have sufficiently addressed the points you raised. Please do not hesitate to let us know if further discussion would be helpful.

Thank you for your continued time and consideration.

Thank you to the authors for the detailed response and additional experiments. I appreciate the efforts to address my concerns, particularly regarding reference-based character identification and evaluator reliability.

The additional results and clarifications have largely addressed my concerns. I believe the paper makes a valuable contribution to story visualization evaluation. Therefore, I will maintain my original positive rating of 3 (Weak Acceptance).

Thank you. We appreciate your valuable insights, and constructive feedback.

Reviewer_L8BwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StoryDynamics, a benchmark for evaluating story visualization across six dimensions: entity count, character identity, character–object interaction, visual state, state transition, and visual quality. Its central design represents narrative states explicitly and evaluates whether their changes occur at the correct narrative steps. The benchmark contains 93 English and Chinese stories with either five or 100 frames and evaluates 21 generation methods. The results highlight weaknesses in character–object interactions and suggest that story-specific methods do not consistently outperform general-purpose models across the evaluated dimensions.

Critical strengths

● The distinction between state correctness and transition timing is useful. Evaluating these separately can identify premature, delayed, or unjustified state changes that character consistency and frame-level text–image alignment may overlook. ● The evaluation includes meaningful long-sequence settings and detailed diagnostics. The 100-frame stories test sustained state tracking, while the range of models, languages, and visual styles provides useful coverage. Property-level results help identify specific model limitations.

Critical weaknesses

● How much character-matching errors affect the other scores? Appendix C.2 uses reference portraits to identify characters before extracting their states and interactions. If the evaluator misses a character who is actually present, this could affect character-count, character-state, and interaction scores together. Have the authors checked how often this happens? A small audit separating identification errors from state and interaction errors would be helpful. ● I would like stronger validation of the benchmark’s central measurements. The human study mainly compares model-level state scores, so how reliably the evaluator detects changes at the wrong narrative step? Likewise, the strongest PViC agreement in Table 10 uses a restricted vocabulary, rather than the story-derived vocabulary used in the main evaluation. Could the authors provide direct checks of these two cases and clarify whether the final state and interaction annotations were independently checked by humans? ● Tables 3 and 13 show differences between groups, but I am concerned how closely those groups were matched in theme, visual style, and linguistic complexity? Were these factors balanced or otherwise controlled? This clarification would help distinguish effects of length and language from differences in the stories being compared.

Questions for authors

see the weakness above

4 comments

Thank you for the thoughtful comments.

Q1: Impact of character-matching errors on downstream evaluation metrics

Character-matching errors affect character-count, character-state, and character-object interaction scores. If a character with a reference image is not recognized in a generated frame, the corresponding character-specific state and interaction scores will also be affected because these properties cannot be correctly attributed to that character.

This design is intentional, as we aim to evaluate whether the correct story character is present together with the corresponding states and interactions, rather than treating character identity and downstream character-specific properties independently. Importantly, character matching is performed separately for each frame, so an identification error in one frame does not propagate to other frames. Characters without reference images are excluded from the scored character-state and interaction sets by design.

Q2: Stronger validation of central measurements; reliability and human validation of narrative state, transition, and interaction metrics

We followed the reviewer's suggestion and conducted additional validation of our central measurements, including character-count, object-count, identity matching, character-object interaction, state and transition scores. We collected both human and VLM judgements on 12 real-world videos, across a total of 72 frames.

We do not conduct validation on quality scores, as we utilize metrics from prior works [1-3], which have demonstrated alignment with human. Meanwhile, the state transition score depends on the state labels, without the need of judgements from evaluators; hence we omit the state transition score from the agreement calculations.

We depict the human-VLM agreement results with Gwet's AC1 [4], alongside both human and VLM results, as follows:

MetricsCntchar↑\mathrm{Cnt}_{char}\uparrowCntobj↑\mathrm{Cnt}_{obj}\uparrowID↑\mathrm{ID}\uparrowF1COI↑\mathrm{F1}_{COI}\uparrowEMCOI↑\mathrm{EM}_{COI}\uparrowStateF1↑\mathrm{StateF1}\uparrowTransF1↑\mathrm{TransF1}\uparrowOverall↑\mathrm{Overall}\uparrow
Human Score0.8520.9000.9220.9130.7670.9800.8050.872
VLM Score0.8470.9310.9270.8610.7750.8200.6840.827
Gwet's AC10.8440.8760.9040.8460.8070.968-0.903

Notably, we observe that across all metrics, both human and VLM judges achieve an overall agreement rate of 0.903, signifying that our central measurements are aligned with human judgements.

Q3: Distributions of groups in Tables 3 and 13; Effects of length and language from differences in groups

We follow the reviewer’s suggestion and clarify the distributions of our benchmark as follows:

Linguistic Complexity

GroupSubgroupComplexSimpleTotal
LengthLong232245
Short331548
LanguageEnglish291645
Chinese272148

Visual Styles

GroupSubgroupAnimeCGIClaymationPhotorealismTotal
LengthLong61591545
Short141491148
LanguageEnglish71481645
Chinese1315101048

We initially constructed the benchmark as a balanced distribution across all themes, linguistic complexities, and visual styles. However, after human annotations and quality filtering, certain categories exhibited higher exclusion rates (e.g., anime, claymation), resulting in some distributional imbalance in the final benchmark.

Although there is imbalance in the distributions, we note that the results in Table 3 is averaged across all evaluated models, providing an aggregated view of performance trends, rather than individual view. We interpret the aggregated results as an empirical proxy to the effect of story lengths and languages.

References:

[1] Wang et al., 2025, Unified Reward Model for Multimodal Understanding and Generation, arXiv'25.

[2] Schuhmann et al., 2022, LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models, NeurIPS'22.

[3] Schuhmann, 2022, LAION-Aesthetics, LAION'22.

[4] Gwet, 2008, Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement, British Journal of Mathematical and Statistical Psychology, 61(1):29–48.

AuthorsOpenReview ↗

Rebuttal Discussion: Follow-Up on Any Remaining Concerns

Dear Reviewer L8Bw,

We would appreciate hearing your thoughts on our response and whether it has addressed your concerns. Please feel free to let us know if any points would benefit from further clarification.

Thank you again for your time and consideration.

Reviewer_L8BwOpenReview ↗

maintain my score

Thanks a lot for the authors response and address some of my concerns, but I maintain my score

Dear Reviewer L8Bw,

Thank you. We appreciate your thoughtful review and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.