- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Omni-StoryBench introduces a benchmark for evaluating multimodal story continuation. Given a current children’s storybook page, book metadata, and structured constraints derived from the next page, a system must generate the next-page illustration, narration, and character speech. The benchmark contains 900 page transitions and is used to compare 32 generation-system configurations. Evaluation combines modality-specific automatic metrics, rubric-based LLM judgments for text, image, speech, and integrated multimodal coherence, and a human study on a fixed subset of 10 transitions. The authors report that orchestrated systems achieve stronger overall scores, while image continuity and the controllability and completeness of native any-to-any systems remain challenging. The study contributes a task, dataset, and evaluation protocol for conditional multimodal story continuation; its human evaluation provides a limited check of agreement between human and LLM-based system rankings.
Critical strengths
A key strength is that the paper operationalizes a difficult-to-evaluate problem—multimodal coherence in generation—as a concrete, measurable benchmark task. Establishing its measurement validity and generalizability would substantially strengthen its potential contribution.
Critical weaknesses
-
The benchmark’s scope and external validity are unclear. Omni-StoryBench evaluates children’s storybook continuation conditioned on the preceding page and explicit constraints derived from the reference next page. This is a useful controlled setting, but the paper does not establish how well success in this oracle-conditioned task reflects broader multimodal generation capabilities. The lack of a systematic analysis across condition difficulty further limits conclusions about what levels of coherence the benchmark can distinguish.
-
The evidence for measurement validity is limited. The aggregate score combines automated metrics and LLM-judge ratings under a fixed weighting scheme, while the human evaluation covers only 10 selected transitions. The paper also reports substantially fewer human ratings than targeted, without explaining the shortfall. Consequently, the reported human–LLM agreement provides limited evidence that the aggregate score robustly measures multimodal coherence across the benchmark.
Questions for authors
-
Clarification: How were the 10 transitions selected for human evaluation? Were they sampled or stratified to represent different sources, condition types, and levels of difficulty? The answer would help establish how broadly the reported human–LLM agreement applies.
-
Evidence relevant to my assessment: Where systems or tasks overlap, could the authors compare Omni-StoryBench results with results on established omnmodal-generation or controlled single-modality generation benchmarks, report the degree of agreement, and analyze important discrepancies? If direct comparisons are not feasible, please clarify what evidence supports the benchmark’s construct validity and what limits remain.
4 comments
Response to Reviewer GbVC
Question 1 : Clarification
The 10 transitions are not a stratified sample. They are a common window obtained by shuffling the order of the 900 transitions with a fixed seed and then selecting, from the front, the transitions for which all 32 configurations had produced text, image, and speech in the generations available at that time (App. F.5 of the submitted version, "pre-selected rather than … formal stratified sampling"). As in SummEval [2], this design has all configurations and tracks evaluated on the same inputs so that system comparisons are not swayed by differences in the transition sample, and the judge rankings on this window also agreed with the rankings on the 900 transitions at ρ 0.84–0.97 (Total 0.948; F.5 of the submitted version).
To broaden the scope of applicability, we conducted an expanded human evaluation of 4,096 ratings with the same common-input design (Table G4). The transitions were newly drawn from the same seed order so as not to overlap with the window, and their mean judge score is 6.58, on par with that of all 900 transitions (6.61). Even when split by source, character change, scene change, emotion, and speech conditions, the human–judge composite rank correlation was 0.92–0.99 in all strata with 5 or more transitions, and it was 0.975/0.966 even in the easier and harder halves by judge score, so the agreement holds across conditions and difficulty levels (see Table G4).
Question 2 : Evidence relevant to my assessment
Because our evaluation purpose differs from that of UniM [3], the degree of rank agreement can hardly serve as a criterion of validity, and indeed there is only one system in common, AnyGPT. For that system in common and for shared trends, the benchmarks are consistent with each other. InterleavedBench, which includes a story task, singled out image coherence (inability to condition on the previous image) as the biggest weakness of pipelines, and this is also consistent with Kontext>dev (10/10 pairs; App. F.8). AnyGPT is last-ranked on our benchmark (32/32, also last-ranked on the human evaluation composite) and, on UniM as well, falls far short of the agent pipeline UniMA (SQCS_abs 11.1–17.9% vs 57.3–72.7%); the trend that planner+expert-generator pipelines score higher than unified models is also the same as in UniM and InterleavedBench. Our benchmark overlaps with single-modality generation benchmarks at the level of generation models (expert image generators and TTS, and native generators such as Emu3 and MMaDA), but their scores measure the quality of a single output for a general prompt, whereas our scores measure story continuity and condition satisfaction, so the interpretation of rank correlations is also limited. Instead, we have compiled the following evidence.
(1) Agreement with single-modality metrics: the image judge has r=−0.910 with the independent perceptual metric DreamSim (App. F.2), and the text judge has Pearson 0.538 and Spearman 0.704 with BERTScore (App. F.1). The main discrepancy is the verbose narration of large VLMs, for which judge scores are high but BERTScore gains are small. We analyzed the cause in Sec 6.2, and the ranking remained nearly unchanged under all 4 length adjustments (Table 28, ρ≥0.998).
(2) Construct validity: we have system-level agreement with the expanded human evaluation (Spearman text 0.965, image 0.976, speech 0.843, integrated 0.979, composite 0.988; Table G4), agreement across the three judge panels (App. F.6), reproduction of expected contrasts (all 12/12 pairs that switch the backbone 4B→large VLM with the same generator show an increase in Total, +0.62~+0.95), and the swap analysis of the integrated evaluator, which judges agreement between the generated outputs (26 configuration pairs in which only one modality's output changes on the same transition; for example, 830 images that copy the current page nearly as is were rated +0.30 higher by the image judge, but the integrated judge penalized them at −0.12 [−0.16, −0.08]; Table G1 & revision App. I).
(3) Remaining scope and complementary measures: we have broadened the scope of the human evaluation with the additional evaluation in our responses to Weakness 2 and Question 1. We intentionally limited the domain to English children's storybooks (Sec 1).
References
References
[1] Wang et al., SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems, NeurIPS 2019. https://proceedings.neurips.cc/paper/2019/hash/4496bf24afe7fab6f046bf4923da8de6-Abstract.html
[2] Fabbri et al., SummEval: Re-evaluating Summarization Evaluation, TACL 2021. https://aclanthology.org/2021.tacl-1.24/
[3] Li et al., UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark, CVPR 2026. https://openaccess.thecvf.com/content/CVPR2026/html/Li_UniM_A_Unified_Any-to-Any_Interleaved_Multimodal_Benchmark_CVPR_2026_paper.html
Dear Reviewer GbVC,
Thank you again for your thoughtful review. Your comments helped us strengthen the evidence for measurement validity and clarify the scope of Omni-StoryBench. We have uploaded the revised manuscript incorporating the changes described in our responses.
Regarding human-evaluation coverage and sampling, we completed all 4,096 planned ratings across 32 configurations and four evaluation tracks, and clarified the transition-selection procedure (Appendix F.5). The human composite ranking agrees with Total Average at Spearman ρ = 0.979, with the same set of top-10 configurations, and with the judge composite at ρ = 0.988. We also report agreement across condition and difficulty strata.
Regarding the fixed weighting scheme, we presented aggregation-sensitivity results supporting the main system-level comparisons. For example, modality-balanced aggregation retains the same top-ranked configuration and yields ρ = 0.995 with the original ranking (Appendix F.8). We additionally included condition-difficulty analyses (Appendix L) and cross-modal swap analyses providing further evidence for the integrated evaluator’s sensitivity to inconsistencies between generated outputs (Appendix I).
We also clarified the benchmark’s intended scope: evaluating whether systems preserve a given story context while consistently realizing a specified next state across image, narration, and speech. Appendix O discusses the rationale for this transition design and its relationship to existing benchmarks.
As the discussion period draws to a close, we would appreciate hearing whether our responses and the revised manuscript address your concerns. Please let us know of any remaining questions so that we can clarify them in the remaining time. We hope you will consider these clarifications and additional evidence in your final assessment.
Thank you for your time and careful consideration.
Response to Reviewer GbVC
Thank you for recognizing the value of turning multimodal coherence—a difficult-to-evaluate problem—into a concrete, measurable benchmark task. Below, we clarify the benchmark’s intended scope and present expanded human evaluation, analyses across conditions and difficulty levels, and additional evidence for measurement validity, while explaining the limits of comparisons with existing benchmarks.
Weakness 1 : The benchmark’s scope and external validity are unclear
We designed Omni-StoryBench not to represent general-purpose multimodal capability as a whole, but to measure, in a controlled setting, the ability to consistently realize a specified next state across the three modalities while preserving the given state (Sec 1, App. A). The conditions derived from the ground-truth page separate 'which event to choose' from 'how to realize that event'. This is because, in open-ended continuation, the ground truth is not uniquely determined, so a realization failure cannot be distinguished from a different choice. As evidence for measurement validity, we have system-level agreement with the expanded human evaluation (ρ=0.988 with the judge composite, 0.979 with Total, and the same paradigm ordering; Table G4) and agreement with an independent perceptual metric (DreamSim–image judge r=−0.910; App. F.2).
We additionally analyzed condition difficulty using existing results. Please refer to Table G3. When stratified by the number of characters visible on the next page (0/1/2/3 or more characters: 32/293/453/122 transitions), image judge scores decrease monotonically in all five paradigms (overall 5.41→5.10→4.92→4.73; likewise for the condition-satisfaction score of Claude-Opus-4.7+Kontext, 6.72 at 1 character→5.90 at 3 or more), and the ranking of the 32 configurations by the image judge is stable across strata (Spearman 0.957–0.994). The integrated score differs little between strata (7.6–7.7), so on this axis it is mainly the image stage that discriminates. Regarding the relationship with broader capabilities as well, the results agree with existing benchmarks at the level of systems in common and shared trends (planner+expert-generator pipelines > unified models; see our response to Question 2). We will add this analysis to the revision.
Weakness 2 : The evidence for measurement validity is limited
We address the three points in turn.
(1) Fixed weights: Total is the equal-weight mean of 7 components, which is the common practice when there is no fair weighting criterion (SuperGLUE [1]). An aggregation that first averages within each modality also yields ρ 0.995 and the same 1st-ranked configuration (Table 28); the ρ between the 18 aggregation variants and the original ranking is 0.905–1.000, with the orchestration group highest in every variant; and even under Dirichlet random weights, the median rank correlation is 0.988 and the rate at which a large VLM+Kontext configuration ranks 1st is 97.7% (App. F.8).
(2) Human evaluation: F.5 in the submitted version was a small-scale evaluation with 687 scored evaluations. This time, we collected 4,096 ratings evaluating the outputs of all 32 configurations (the same files as in the main results) in 4 tracks (see Table G4). The results of the expanded human evaluation experiment show that the human composite ranking has ρ=0.979 with Total (τ=0.910, top 10: 10/10) and ρ=0.988 with the judge composite, and that the correlation remains at 0.92–0.99 in the character, scene, emotion, and speech condition strata as well. In the revision, we will replace F.5 with this evaluation. (3) Other evidence: image judge–DreamSim r=−0.910 (App. F.2), reproduction of expected contrasts (Kontext>dev in 10/10 pairs, 4B→large backbone in 12/12 pairs), and swaps that change only one modality (830 images that nearly copy the current page: image judge +0.30 vs integrated judge −0.12; Table G1).