acceptodds
Under review as a conference paper at ICLR 2027

What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generation

Abstract

Omnimodal evaluation should go beyond independent text, image, and speech production: individually plausible outputs may not express a coherent shared event. We introduce Omni-StoryBench, a story-grounded omnimodal benchmark evaluating whether models can coherently continue stories across image, narration, and speech. Each instance provides a current storybook page and structured next-page conditions, requiring models to generate the next illustration, narration, and spoken character utterance. Omni-StoryBench contains 900 rigorously validated story transitions from openly licensed children's books, with ground-truth next-page references and speech metadata. We evaluate systems with modality-specific metrics and consistency-centered LLM-as-a-judge rubrics for context preservation, condition following, reference consistency, and cross-modal coherence. Across 32 baseline configurations spanning orchestration, semi-orchestration, and native any-to-any paradigms, we find orchestration with strong VLM planning most reliable, while current native omnimodal models often struggle with output completeness and controllability. Our analysis shows text-side performance is associated with image and speech quality, but image generation and visual continuity form the clearest observed bottleneck among the evaluated configurations. These results position Omni-StoryBench as a system-level benchmark measuring coherent omnimodal generation beyond isolated modality quality.

Primary area:
datasets and benchmarks
Keywords:
Omnimodal generation, dataset, story continuation, cross-modal consistency, text-image-speech generation, multimodal generation, multimodal evaluation, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GbVCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Omni-StoryBench introduces a benchmark for evaluating multimodal story continuation. Given a current children’s storybook page, book metadata, and structured constraints derived from the next page, a system must generate the next-page illustration, narration, and character speech. The benchmark contains 900 page transitions and is used to compare 32 generation-system configurations. Evaluation combines modality-specific automatic metrics, rubric-based LLM judgments for text, image, speech, and integrated multimodal coherence, and a human study on a fixed subset of 10 transitions. The authors report that orchestrated systems achieve stronger overall scores, while image continuity and the controllability and completeness of native any-to-any systems remain challenging. The study contributes a task, dataset, and evaluation protocol for conditional multimodal story continuation; its human evaluation provides a limited check of agreement between human and LLM-based system rankings.

Critical strengths

A key strength is that the paper operationalizes a difficult-to-evaluate problem—multimodal coherence in generation—as a concrete, measurable benchmark task. Establishing its measurement validity and generalizability would substantially strengthen its potential contribution.

Critical weaknesses
  1. The benchmark’s scope and external validity are unclear. Omni-StoryBench evaluates children’s storybook continuation conditioned on the preceding page and explicit constraints derived from the reference next page. This is a useful controlled setting, but the paper does not establish how well success in this oracle-conditioned task reflects broader multimodal generation capabilities. The lack of a systematic analysis across condition difficulty further limits conclusions about what levels of coherence the benchmark can distinguish.

  2. The evidence for measurement validity is limited. The aggregate score combines automated metrics and LLM-judge ratings under a fixed weighting scheme, while the human evaluation covers only 10 selected transitions. The paper also reports substantially fewer human ratings than targeted, without explaining the shortfall. Consequently, the reported human–LLM agreement provides limited evidence that the aggregate score robustly measures multimodal coherence across the benchmark.

Questions for authors
  1. Clarification: How were the 10 transitions selected for human evaluation? Were they sampled or stratified to represent different sources, condition types, and levels of difficulty? The answer would help establish how broadly the reported human–LLM agreement applies.

  2. Evidence relevant to my assessment: Where systems or tasks overlap, could the authors compare Omni-StoryBench results with results on established omnmodal-generation or controlled single-modality generation benchmarks, report the degree of agreement, and analyze important discrepancies? If direct comparisons are not feasible, please clarify what evidence supports the benchmark’s construct validity and what limits remain.

4 comments
AuthorsOpenReview ↗

Response to Reviewer GbVC

Thank you for recognizing the value of turning multimodal coherence—a difficult-to-evaluate problem—into a concrete, measurable benchmark task. Below, we clarify the benchmark’s intended scope and present expanded human evaluation, analyses across conditions and difficulty levels, and additional evidence for measurement validity, while explaining the limits of comparisons with existing benchmarks.

Weakness 1 : The benchmark’s scope and external validity are unclear

We designed Omni-StoryBench not to represent general-purpose multimodal capability as a whole, but to measure, in a controlled setting, the ability to consistently realize a specified next state across the three modalities while preserving the given state (Sec 1, App. A). The conditions derived from the ground-truth page separate 'which event to choose' from 'how to realize that event'. This is because, in open-ended continuation, the ground truth is not uniquely determined, so a realization failure cannot be distinguished from a different choice. As evidence for measurement validity, we have system-level agreement with the expanded human evaluation (ρ=0.988 with the judge composite, 0.979 with Total, and the same paradigm ordering; Table G4) and agreement with an independent perceptual metric (DreamSim–image judge r=−0.910; App. F.2).

We additionally analyzed condition difficulty using existing results. Please refer to Table G3. When stratified by the number of characters visible on the next page (0/1/2/3 or more characters: 32/293/453/122 transitions), image judge scores decrease monotonically in all five paradigms (overall 5.41→5.10→4.92→4.73; likewise for the condition-satisfaction score of Claude-Opus-4.7+Kontext, 6.72 at 1 character→5.90 at 3 or more), and the ranking of the 32 configurations by the image judge is stable across strata (Spearman 0.957–0.994). The integrated score differs little between strata (7.6–7.7), so on this axis it is mainly the image stage that discriminates. Regarding the relationship with broader capabilities as well, the results agree with existing benchmarks at the level of systems in common and shared trends (planner+expert-generator pipelines > unified models; see our response to Question 2). We will add this analysis to the revision.

Weakness 2 : The evidence for measurement validity is limited

We address the three points in turn.

(1) Fixed weights: Total is the equal-weight mean of 7 components, which is the common practice when there is no fair weighting criterion (SuperGLUE [1]). An aggregation that first averages within each modality also yields ρ 0.995 and the same 1st-ranked configuration (Table 28); the ρ between the 18 aggregation variants and the original ranking is 0.905–1.000, with the orchestration group highest in every variant; and even under Dirichlet random weights, the median rank correlation is 0.988 and the rate at which a large VLM+Kontext configuration ranks 1st is 97.7% (App. F.8).

(2) Human evaluation: F.5 in the submitted version was a small-scale evaluation with 687 scored evaluations. This time, we collected 4,096 ratings evaluating the outputs of all 32 configurations (the same files as in the main results) in 4 tracks (see Table G4). The results of the expanded human evaluation experiment show that the human composite ranking has ρ=0.979 with Total (τ=0.910, top 10: 10/10) and ρ=0.988 with the judge composite, and that the correlation remains at 0.92–0.99 in the character, scene, emotion, and speech condition strata as well. In the revision, we will replace F.5 with this evaluation. (3) Other evidence: image judge–DreamSim r=−0.910 (App. F.2), reproduction of expected contrasts (Kontext>dev in 10/10 pairs, 4B→large backbone in 12/12 pairs), and swaps that change only one modality (830 images that nearly copy the current page: image judge +0.30 vs integrated judge −0.12; Table G1).

AuthorsOpenReview ↗

Response to Reviewer GbVC

Question 1 : Clarification

The 10 transitions are not a stratified sample. They are a common window obtained by shuffling the order of the 900 transitions with a fixed seed and then selecting, from the front, the transitions for which all 32 configurations had produced text, image, and speech in the generations available at that time (App. F.5 of the submitted version, "pre-selected rather than … formal stratified sampling"). As in SummEval [2], this design has all configurations and tracks evaluated on the same inputs so that system comparisons are not swayed by differences in the transition sample, and the judge rankings on this window also agreed with the rankings on the 900 transitions at ρ 0.84–0.97 (Total 0.948; F.5 of the submitted version).

To broaden the scope of applicability, we conducted an expanded human evaluation of 4,096 ratings with the same common-input design (Table G4). The transitions were newly drawn from the same seed order so as not to overlap with the window, and their mean judge score is 6.58, on par with that of all 900 transitions (6.61). Even when split by source, character change, scene change, emotion, and speech conditions, the human–judge composite rank correlation was 0.92–0.99 in all strata with 5 or more transitions, and it was 0.975/0.966 even in the easier and harder halves by judge score, so the agreement holds across conditions and difficulty levels (see Table G4).

Question 2 : Evidence relevant to my assessment

Because our evaluation purpose differs from that of UniM [3], the degree of rank agreement can hardly serve as a criterion of validity, and indeed there is only one system in common, AnyGPT. For that system in common and for shared trends, the benchmarks are consistent with each other. InterleavedBench, which includes a story task, singled out image coherence (inability to condition on the previous image) as the biggest weakness of pipelines, and this is also consistent with Kontext>dev (10/10 pairs; App. F.8). AnyGPT is last-ranked on our benchmark (32/32, also last-ranked on the human evaluation composite) and, on UniM as well, falls far short of the agent pipeline UniMA (SQCS_abs 11.1–17.9% vs 57.3–72.7%); the trend that planner+expert-generator pipelines score higher than unified models is also the same as in UniM and InterleavedBench. Our benchmark overlaps with single-modality generation benchmarks at the level of generation models (expert image generators and TTS, and native generators such as Emu3 and MMaDA), but their scores measure the quality of a single output for a general prompt, whereas our scores measure story continuity and condition satisfaction, so the interpretation of rank correlations is also limited. Instead, we have compiled the following evidence.

(1) Agreement with single-modality metrics: the image judge has r=−0.910 with the independent perceptual metric DreamSim (App. F.2), and the text judge has Pearson 0.538 and Spearman 0.704 with BERTScore (App. F.1). The main discrepancy is the verbose narration of large VLMs, for which judge scores are high but BERTScore gains are small. We analyzed the cause in Sec 6.2, and the ranking remained nearly unchanged under all 4 length adjustments (Table 28, ρ≥0.998).

(2) Construct validity: we have system-level agreement with the expanded human evaluation (Spearman text 0.965, image 0.976, speech 0.843, integrated 0.979, composite 0.988; Table G4), agreement across the three judge panels (App. F.6), reproduction of expected contrasts (all 12/12 pairs that switch the backbone 4B→large VLM with the same generator show an increase in Total, +0.62~+0.95), and the swap analysis of the integrated evaluator, which judges agreement between the generated outputs (26 configuration pairs in which only one modality's output changes on the same transition; for example, 830 images that copy the current page nearly as is were rated +0.30 higher by the image judge, but the integrated judge penalized them at −0.12 [−0.16, −0.08]; Table G1 & revision App. I).

(3) Remaining scope and complementary measures: we have broadened the scope of the human evaluation with the additional evaluation in our responses to Weakness 2 and Question 1. We intentionally limited the domain to English children's storybooks (Sec 1).

AuthorsOpenReview ↗

References

References

[1] Wang et al., SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems, NeurIPS 2019. https://proceedings.neurips.cc/paper/2019/hash/4496bf24afe7fab6f046bf4923da8de6-Abstract.html

[2] Fabbri et al., SummEval: Re-evaluating Summarization Evaluation, TACL 2021. https://aclanthology.org/2021.tacl-1.24/

[3] Li et al., UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark, CVPR 2026. https://openaccess.thecvf.com/content/CVPR2026/html/Li_UniM_A_Unified_Any-to-Any_Interleaved_Multimodal_Benchmark_CVPR_2026_paper.html

Dear Reviewer GbVC,

Thank you again for your thoughtful review. Your comments helped us strengthen the evidence for measurement validity and clarify the scope of Omni-StoryBench. We have uploaded the revised manuscript incorporating the changes described in our responses.

Regarding human-evaluation coverage and sampling, we completed all 4,096 planned ratings across 32 configurations and four evaluation tracks, and clarified the transition-selection procedure (Appendix F.5). The human composite ranking agrees with Total Average at Spearman ρ = 0.979, with the same set of top-10 configurations, and with the judge composite at ρ = 0.988. We also report agreement across condition and difficulty strata.

Regarding the fixed weighting scheme, we presented aggregation-sensitivity results supporting the main system-level comparisons. For example, modality-balanced aggregation retains the same top-ranked configuration and yields ρ = 0.995 with the original ranking (Appendix F.8). We additionally included condition-difficulty analyses (Appendix L) and cross-modal swap analyses providing further evidence for the integrated evaluator’s sensitivity to inconsistencies between generated outputs (Appendix I).

We also clarified the benchmark’s intended scope: evaluating whether systems preserve a given story context while consistently realizing a specified next state across image, narration, and speech. Appendix O discusses the rationale for this transition design and its relationship to existing benchmarks.

As the discussion period draws to a close, we would appreciate hearing whether our responses and the revised manuscript address your concerns. Please let us know of any remaining questions so that we can clarify them in the remaining time. We hope you will consider these clarifications and additional evidence in your final assessment.

Thank you for your time and careful consideration.

Reviewer_r4roOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Omni-StoryBench, a 900-sample benchmark built from openly licensed children's storybooks to evaluate omnimodal (image, narration text, and speech) generation. Each test instance is a single two-page transition: conditioned on one current-page image, current narration, book metadata, and a detailed next-page prompt (generated by Qwen3-VL-32B from the ground-truth next page), models are asked to generate the next illustration, narration, and character speech. The authors evaluate 32 system configurations (spanning VLM orchestration, semi-orchestration, and native any-to-any models) using automatic metrics and three-judge open-weight LLM/VLM/ALM ensembles, concluding that VLM orchestration outperforms current open-source any-to-any models and that visual continuity remains the main bottleneck.

Critical strengths
  1. Extensive evaluation pipeline and statistical analysis. The authors put solid effort into building a reproducible evaluation harness, combining automatic metrics with three-judge ensembles across text, image, speech, and integrated tracks, backed by thorough statistical checks in the appendix (such as deduplication audits, zero-filled vs. valid-only comparisons, bootstrap intervals, and human evaluation agreement).

  2. Useful diagnostics on formatting stability and judge verbosity bias. Tracking missing-modality and JSON orchestration errors separately from generation quality (Figure 4) clearly shows where current semi-orchestration and any-to-any pipelines break down, and Section 6.2 provides a helpful observation on how LLM judges favor overly verbose narrations over concise storybook style.

Critical weaknesses
  1. Awkward positioning as a "story" benchmark and questionable speech labels. As discussed in Related Works, cross-modal consistency in any-to-any interleaved generation has already been studied by benchmarks such as UniM. While this paper claims to focus on story continuation ("What Comes Next?"), every sample only conditions on a single current image and a detailed Next-page Condition prompt (which is generated by Qwen3-VL-32B looking directly at the ground-truth next page and accounts for >80% of input words). This reduces the task to single-step conditional rendering rather than multi-step story generation. Furthermore, because the source storybooks contain no audio, both the spoken dialogue lines and the four coarse metadata labels (emotion, speed, pitch, binary gender) are synthetically generated by an LLM. As the paper's own analysis shows (Appendix A, Table 23), audio judges have very low item-level agreement (0.11-0.35) and speech scores barely separate different models.

  2. Plain empirical conclusions that offer limited new insight for advancing SOTA. Although the statistical reporting is extensive, most of the 32 baselines are combinatorial pipelines (5 VLMs x 3 image generators x 2 TTS), and the main empirical conclusions—that orchestrating large closed-source VLMs (Claude-Opus-4.7, GPT-5.4) with specialist image/TTS models beats small 7B/8B open-source any-to-any models, and that feeding the reference image into FLUX.1-Kontext improves visual continuity over text-only FLUX.1-dev—are not surprising. The paper would be much stronger if it went beyond these expected pipeline comparisons to uncover non-trivial failure modes of SOTA systems that actively guide future model design.

Questions for authors
  1. Single-step prompt following vs. story continuation: Since each sample only spans one page transition and the input prompt explicitly describes the ground-truth next page, how does Omni-StoryBench evaluate genuine story continuation rather than single-step instruction following?
  2. Reliability of the synthetic speech track: Given that the speech utterances and attributes are LLM-generated without ground-truth audio, and audio judges show very low item-level correlation (0.11-0.35 in Table 23), how can the speech score be made more discriminative and reliable?
  3. Actionable failure cases of SOTA systems: Beyond showing that frontier VLM orchestration beats small any-to-any models and that image conditioning helps visual continuity, what specific, non-trivial failure cases do even the strongest SOTA configurations (e.g., Claude-Opus-4.7 / GPT-5.4 + FLUX.1-Kontext) exhibit on Omni-StoryBench that can guide future progress in the field?
9 comments
AuthorsOpenReview ↗

Response to Reviewer r4ro

Thank you for recognizing the extensive evaluation pipeline and statistical analyses, as well as the useful diagnostics on formatting stability, missing modalities, and judge verbosity bias. Below, we clarify the purpose of the single-step task and present expanded human listening evaluation, speech-content diagnostics, and concrete failure cases that persist even in the strongest configurations.

Weakness 1 : Awkward positioning as a "story" benchmark

We agree that UniM already addresses cross-modal consistency in any-to-any interleaved generation, and our paper also states this (Sec 1–2). The difference between the two benchmarks lies in what is held fixed and what is judged. UniM [1] evaluates responses in various modality combinations on open-form QA across 30 domains, and ICS converts non-text outputs into captions and then judges holistic coherence (HC) and stylistic harmony (SH) (UniM §4.3, App. D.3.3). Omni-StoryBench fixes the current-page state and a specified next state, and judges whether that state is consistently realized in the image, narration, and utterance, in light of the actual image and speech and the ground-truth next page (App. E.4.4). Therefore, it is appropriate neither to view Omni-StoryBench as 'a smaller-scale UniM' nor to view UniM as 'a benchmark that cannot evaluate story consistency', and the two benchmarks are complementary to each other. The single-transition design is a deliberate choice made for this kind of control. Because the conditions fix 'what happens', failures can be attributed not to plot choices but to problems of visual state preservation and cross-modal realization. Also, the next-page conditions have no appearance field, and appearance information in the book metadata is also rare (about 15% of the characters in the conditions), so appearance and art style must mostly be carried over from the current image; the fact that Kontext, which takes the current image as input, scores higher than dev in 10/10 pairs is also consistent with this (App. F.8).

As you noted, the task is single-step: each instance is a single transition between two pages (App. C), and we deliberately restricted the scope to one-page continuation (Sec 1, "intentionally constrained scope"). If a story is viewed as a chain of state transitions (a Markov approximation), the current page (image and narration) and the book-level metadata built from the book's full narration (genre, topic, narrative perspective, and character profiles; Sec 4.1) are the state, and the next-page conditions are the intended event. Omni-StoryBench evaluates a one-step transition starting from the ground-truth current state (teacher forcing). If multiple steps are chained on top of the model's own outputs, an error at one step (e.g., a change in a character's appearance) accumulates into the next steps, making it difficult to tell at which step and in which modality a failure occurred. Therefore, single-step evaluation is not a substitute for multi-step evaluation but the unit diagnostic it presupposes. Current systems have not yet solved even this unit transition. Even the best configuration scores 6.81/10 on the image judge, and GT semantic consistency in the integrated evaluation is the lowest criterion in all 32 configurations (Table 11). In the any-to-any and semi-orchestration families, missing modalities and format errors are frequent even on a single page (Fig. 4), and 3 of the 4 any-to-any models were invoked separately per output modality (Sec 5.1). For example, for a system with a speech missing rate of 12.8% (115/900), assuming independence, the probability of at least one omission over 5 pages is about 50%, so in multi-step evaluation, omissions rather than continuity are likely to determine the results. We will explicitly state multi-step evaluation using the source data (11,434 books, 91,449 transitions) as a limitation and future work.

AuthorsOpenReview ↗

Response to Reviewer r4ro

Weakness 1 : questionable speech labels

As you noted, the GT speech is not audio but a textual specification of the dialogue line, speaker, emotion, speed, pitch, and gender, and this is a design intended for automatic evaluation (Sec 3). About 30% of the dialogue lines use the original sentences of the ground-truth next page verbatim (for pages with double-quoted dialogue, about 98% use that dialogue), and the remaining dialogue lines and the four attribute labels of all transitions were created by an LLM and then carefully checked in full by 16 human reviewers and revised where necessary (App. D, Sec 4.1). We also instructed the judges to use the GT as a reference for meaning and persona rather than for surface wording (App. E.4.3).

When we re-analyze the scores of the experimental results from before the revision, although item-level agreement is low (ρ 0.11–0.35; Table 23), the configuration-level speech scores are stable with respect to the transition sample and discriminative. The correlation between the two rankings obtained by splitting the 900 transitions in half is 0.98, 82% of the 153 pairs of the 18 speech systems formed by grouping configurations that produce the same speech are significantly distinguished under the same Holm correction as in F.7, and even counting only the pairs in which all three judges point in the same direction, the share is 60%.

Also, for this revision, we further expanded the scale of the evaluation and conducted a human listening evaluation. As a result, we confirmed that the ranking is preserved with high agreement (ρ=0.84; please see Table G4). The differing score levels across judges (means of 8.29/5.87/4.66) are in line with prior results (AudioJudge [2]) showing that the non-lexical judgments of audio LLM judges are still unstable, so we average the three judges. When the three judges are averaged, the ranking of the 32 baselines agreed with that of the expanded human listening evaluation of 1,024 speech samples that we conducted, at ρ=0.84 (7 of the top 10 and the last-ranked AnyGPT are the same).

AuthorsOpenReview ↗

Response to Reviewer r4ro

Weakness 2 : Plain empirical conclusions that offer limited new insight for advancing SOTA

It is true that large-VLM orchestration scores the highest, but our conclusion is not that 'it is better because it is a large closed-source VLM'. Sec 5.2 explicitly stated that this trend should not be interpreted in terms of backbone scale alone, and the 6 configurations of 4B orchestration come out ahead in 59 of the 84 pairwise comparisons with the 14 semi-orchestration and any-to-any configurations (App. F.8). GLM-4.6V in the large group is also open-weight.

The baselines also include small combinations whose parameter scale is similar to that of any-to-any. Please refer to Table G2. The Qwen3.5-4B+VoxCPM2, InternVL3.5-4B+Parler-large, Qwen2.5-Omni-7B, and EMOVA-7B combinations that use Nitro-T-1.2B have total parameters, including encoders, VAEs, and vocoders, of 9.85–13.54B, similar to any-to-any (8.35B–13.27B or more). They outperform 3 of the 4 any-to-any models in Total (6.40–6.60 vs 4.46–6.20; exception: Dynin-Omni, 6.94), and outperform all 4 on the integrated judge (7.19–7.70 vs ≤6.95).

The image-bottleneck conclusion also holds at small scale. With the same 4–7B backbone, the change in Total from switching only the image generator (Nitro-T→FLUX.1-Kontext +0.43–0.56, and the same-size FLUX.1-dev→Kontext +0.15–0.28) is larger than that in the 8 TTS-only pairs (|ΔTotal|<0.12).

Likewise, the point of the comparison between Kontext and dev is to quantify the visual state that is lost when the same plan is passed on only as text (image judge continuity +0.73 in the 10 pairs whose plans are identical by md5; a natural experiment). Furthermore, even in the top-ranked configuration, failures recur in which the narration is verbose, the plan is right but the image is wrong (15.9%; including cases of redrawing the current-page character), or the speech is not rendered as planned; these are summarized in our response to Question 3.

Reviewer_J9UuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Omni-StoryBench evaluates whether a system can generate a coherent next story page with an illustration, narration, and character speech. It contains 900 page transitions from openly licensed children’s books. Each input includes the current page and structured conditions describing the expected next page. The paper combines reference-based metrics with modality-specific and integrated model judges. It evaluates 32 configurations covering orchestration, semi-orchestration, and native any-to-any systems. The main findings are that strong orchestration systems perform better, while visual continuity and output completeness remain difficult.

Critical strengths
  1. Evaluating whether image, narration, and speech express the same story state is a useful problem. The page references and structured conditions make the task more concrete than unrestricted multimodal generation.
  2. The evaluation work is substantial. The paper includes several judge families, independent human ratings, missing-output analysis, duplicate checks, bootstrap intervals, and aggregation sensitivity. These checks give useful support for the broad system comparisons.
Critical weaknesses
  1. The additional value over existing benchmarks is not sufficiently demonstrated. UniM already evaluates any-to-any generation and interleaved coherence, while InterleavedBench studies image and text coherence. The story setting and character speech are useful additions, but the paper needs clearer evidence of what new failures or system differences they reveal.
  2. The dependence on the current story page is not tested. The next-page conditions already describe characters, actions, emotions, and the scene, and are constructed using the reference next page. Without removing or shuffling the current image and narration, it is unclear how much the task requires story continuity beyond following these conditions.
  3. The integrated evaluator needs more direct validation of cross-modal inconsistency. System-level agreement with humans is useful, but it does not show whether the evaluator can distinguish three individually good outputs from three outputs that agree with each other. Controlled image, narration, or speech swaps would test this central requirement.
  4. Speech evaluation remains a weak part of the benchmark. Human agreement is low, with ordinal alpha of 0.26–0.35. The automatic metadata score checks only four coarse attributes and does not directly measure spoken-content correctness or intelligibility. The speech judges cover more aspects, but their reliability needs stronger listening-based validation.
  5. The headline score is mainly driven by model judgments. Appendix F.8 reports that the four judge components account for 86.4% of its observed variance. The sensitivity analysis is helpful, but automatic-only scoring moves Dynin-Omni from 15th to first.
  6. The 32 configurations provide broad coverage, but they do not isolate why one paradigm works better. Backbones, generators, model sizes, and invocation procedures differ. The results establish performance for these systems, while giving less explanation of the benefit from joint generation or explicit orchestration. Cost and latency are also not measured.
  7. The image-bottleneck analysis is descriptive. Score dispersion, relative ranks, and partial correlations do not show how much correcting image errors would improve the joint output.

After rebuttal: I raise my rating from 2 to 3. The swap tests show that the integrated judge catches some mismatches that the individual judges miss, and the expanded human listening study makes the speech rankings more convincing. My main remaining concern is how much the task depends on the current page.

Questions for authors
  1. How much do the scores change when the current-page image or narration is removed or shuffled, while the next-page conditions remain unchanged?
  2. Does human agreement remain similar on a larger sample covering different character changes, scene changes, emotional transitions, and speech conditions?
  3. What additional conclusions does this benchmark provide when the same systems are also evaluated on the closest existing benchmarks?
9 comments
AuthorsOpenReview ↗

Response to Reviewer J9Uu

Thank you for recognizing the value of evaluating whether images, narration, and speech express the same story state, the concrete task formulation using page references and structured conditions, and the substantial evaluation effort supporting our system comparisons. We address your concerns below with additional analyses, expanded human evaluation, and latency and GPU memory measurements.

Weakness 1

As you noted, UniM evaluates the coherence of any-to-any interleaved generation and InterleavedBench evaluates image–text coherence, as our paper also states (Sec 1–2). Omni-StoryBench differs from them in its evaluation unit and evaluation inputs. (i) Given the current page (image and narration) and a specified next state, it requires systems to jointly generate the image, narration, and character utterance, and it evaluates continuity with the current scene, condition satisfaction, and semantic consistency with the ground-truth next page together. (ii) The ICS of UniM [1] converts non-text outputs into captions and judges them using text only (UniM App. D.3.3; generation quality (GQ) is evaluated separately), whereas our integrated evaluator directly takes the actual images and the generated speech as input (App. E.3.1; see our response to Weakness 3 on this evaluator responding to inconsistency between the generated outputs). The failures and system differences revealed in this setting are as follows.

  • Semantic agreement with the ground-truth next state: In the integrated evaluation, semantic consistency with the ground-truth next page (GT consistency) is the lowest of the four criteria in all 32 configurations (mean 6.91; continuity 7.59, condition satisfaction 7.61, metadata alignment 8.26; Table 11).
  • Verbosity: Large VLMs write narration about twice as long as the ground truth (about 1.8 times when excluding dialogue), but the judges barely penalize this (Sec 6.2).
  • Speech attributes: Even in the top-ranked configurations, the planned emotions and speaking speeds such as 'fast' are hardly realized in the rendering (classifier-based emotion κ 0.10–0.13; the actual speed for 'fast' plans is nearly the same as for 'normal'; gender is mostly realized).
  • Visual state preservation: FLUX.1-Kontext, which takes the current image as input, scores higher than FLUX.1-dev in all 10 matched pairs (App. F.8).

We have added specific failure cases of large-VLM orchestration (the top-ranked configurations) to revision App. K, together with example images.

Weakness 2

As you noted, we constructed the conditions by looking at the current page and the ground-truth next page in order to fix 'what happens' (Sec 4.1; App. D). However, we tested dependence on the current image through the image-generation-stage comparison (Sec 5.1; see our response to Question 1). We designed our task to measure the ability to consistently realize a specified next state in the three modalities while preserving the given context (Sec 1), and we score continuity with the current page as a criterion separate from condition satisfaction (App. E.4.1, E.4.2, E.4.4). Indeed, the scores are not explained by condition following alone.

  • Information not in the conditions: There is no character-appearance field (Sec 4.1; Fig. 8–9), about 15% of the characters in the conditions have appearance information in the book metadata, and for about 70% of the transitions, neither the character conditions nor the profiles contain any description of color, clothing, or hair.
  • Dependence in the ground truth itself: The HSV color-histogram intersection between the ground-truth next illustration and the current illustration is 0.54, higher than that with the current illustrations of other books (0.15), and 26.6% of the ground-truth narrations reuse content words of the current narration that are not in the conditions or metadata (5.8% for other books; 853 transitions).
  • Judge rationales: 45.5% and 33.9% of the continuity rationales of the text and integrated judges, respectively (2.6% and 2.2% when the same rationales are checked against the narrations of other books), mentioned words that appear only in the current narration (not in the judge's other text inputs), and in the condition-satisfaction rationales the figures were 1.5% and 1.4%.
  • Responses by intervention: On the same transition, switching the planner from 4B to a large VLM raises the integrated judge's condition satisfaction by 0.39 more than its continuity, and switching to an image generator that receives the current image raises continuity by 0.04 more than condition satisfaction (difference between the interventions 0.43 [95% CI 0.39–0.47]).
  • Copying is penalized: The 830 cases that nearly copy the current page were rated higher by the image judge relative to dev (+0.30), but the integrated judge penalized them (−0.12; see our response to Weakness 3).

We will add this analysis to the revision.

AuthorsOpenReview ↗

Response to Reviewer J9Uu

Weakness 3

Since the single-modality judges do not see the generated outputs of other modalities (App. E.4), only the integrated judge can assess inconsistency between the generated outputs. To verify this, we performed a swap analysis on 26 configuration pairs in which, on the same transition, the other two output files are exactly identical and only the output of one modality is replaced with that of a different generator. Please refer to Table G1.

  • Image swap: In the 18 pairs in which only the image changed, the integrated score moved in the same direction as the image judge in 76% of the 13,732 cases in which both scores changed (r=0.62).
  • Distinguishing individual quality from agreement: Images that copy the current page nearly as is (830 cases) were, on the contrary, rated higher by the image judge (+0.30), but all three integrated judges penalized them (−0.12 [−0.16, −0.08]). On continuity, the directions were exactly opposite, at +0.70 for the image judge and −0.15 for the integrated judge, and the integrated judge assessed, for example, that "The generated narration and speech are forward-looking, but the image fails to show the action …, creating a disconnect between modalities".
  • Speech swap: In the 8 pairs in which only the TTS changed (6,813 cases), the change in the integrated score co-varied with the degree to which the actually audible dialogue line (Whisper transcript) matches the narration (coefficient +0.36, t=7.5), and accounting for this agreement, the change in the speech judge score was not significant (t=1.7). When the dialogue line was rendered hard to understand, the integrated score decreased (−0.16 [−0.20, −0.13]).

That is, the integrated evaluator judges not only the quality of individual outputs but also the agreement between the generated outputs, and it responds to inconsistencies that the individual judges cannot see. We will add this swap analysis to the revision.

Weakness 4

We agree that speech is the most difficult track, and we have addressed the three points as follows.

  • Listening-based evaluation: We have substantially expanded the human evaluation. Humans listened to and evaluated 1,024 speech samples from all 32 configurations (the same speech as in the main experiments). Please see Table G4. When we substantially expanded the human evaluation, we obtained higher agreement than before. The speech ranking by human scores correlates with the judge ranking on 900 transitions at ρ=0.84 (τ=0.65) and shares 7 of the top 10 and the last-ranked configuration (AnyGPT) with it, and for the 18 speech systems formed by grouping configurations that produce the same speech, ρ=0.90. The α of 0.26–0.35 in F.5 of the submitted version is the item-level inter-annotator agreement of the small-scale evaluation (4.7 ratings per speech cell), and speech scores are compared at the configuration level.
  • Spoken content and intelligibility: When word accuracy (1−min(1, WER)) is computed from the planned dialogue lines and the Whisper-large-v3 transcripts of the generated speech, the 28 configurations with planned dialogue lines have means of 0.83–0.98, and utterances at or below 0.4 make up 0.3–11.4%. The speech judge gives such utterances lower scores on average than utterances at or above 0.8, but it does not directly measure content errors, so we will add this metric as a separate diagnostic metric.
  • Speech Metadata Accuracy: It is one of the 7 components, and even when it is removed, the rank correlation is ρ=0.994 (Table 28). We instructed the speech judge to score spoken content, intelligibility, emotion, prosody, and persona based on the actually audible speech (App. E.4.3).
AuthorsOpenReview ↗

Response to Reviewer J9Uu

Weakness 5

The figures you cite (the 86.4% variance share of the 4 judge components, and Dynin-Omni moving 15th→1st under automatic-only) are correct (App. F.8). However, we do not believe this means that the headline score is fragile.

First, the 86.4% reflects not the weights but differences in between-system discriminative power. The seven components have equal weights, but the between-system standard deviations of the automatic metrics are 0.15–0.60, all smaller than those of the judges (0.68–1.82) (Table 27). Even with per-component z-score standardization, the ranking is nearly the same (ρ=0.990), and the 1st place is also the same.

Second, the headline score is robust to the aggregation method. In 17 of the 18 variants, the 1st place is a large VLM+FLUX.1-Kontext configuration (ρ≥0.981), and even under random weights, the median correlation with the original ranking is 0.988 (Table 28, App. F.8).

Third, automatic-only is an exception caused by a single system. Excluding Dynin-Omni, the automatic-only ranking also shows a high correlation with the original ranking (ρ=0.943), and the 2nd–9th places are all large orchestration. Dynin-Omni is 1st on all three automatic metrics but 13th–25th on the four judges.

Fourth, the expanded human evaluation (Table G4) also supports the headline. The human composite ranking has ρ=0.979 with Total and ρ=0.84 with automatic-only, and Dynin-Omni, which is 1st under automatic-only, is 22nd by human scores (15th in Total). Because automatic metrics only look at similarity to a single ground truth or agreement on coarse speech attributes, they cannot sufficiently judge condition satisfaction or cross-modal consistency. In the revision, we will present the automatic-only and judge-only rankings together with a per-component table and add interpretation guidelines.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (1/3)

Table G1. Integrated-evaluator swap analysis (change in judge scores, 1–10 scale; [ ] denotes the 95% bootstrap CI)

Swap (the other two output files are identical)SampleResult
Image only swapped (18 configuration pairs)13,732 cases in which both the image and integrated judge scores changedintegrated judge change in the same direction as the image judge change in 76% of cases (r=0.62)
└ Images that nearly copy the current page (vs. the FLUX.1-dev images from the same plan)830 casesimage judge +0.30 (continuity +0.70) vs integrated judge −0.12 [−0.16, −0.08] (continuity −0.15); all 3 integrated judge models penalize them
TTS only swapped (8 configuration pairs)6,813 casesintegrated judge change co-varies with the audible line–narration agreement (coefficient +0.36, t=7.5); controlling for this agreement, the speech judge change is not significant (t=1.7)
└ Dialogue line rendered hard to understand339 casesintegrated judge −0.16 [−0.20, −0.13]

Table G2. Comparison of small pipelines of similar parameter scale (using Nitro-T-1.2B) with any-to-any

SystemGeneration modeTotal parameters*Total (rank)Judge meanIntegrated judge
InternVL3.5-4B+Nitro-T+Parler-largeorchestration9.87B6.40 (25)5.977.19
Qwen3.5-4B+Nitro-T+VoxCPM2orchestration9.85B6.57 (23)6.277.43
EMOVA-7B+Nitro-Tsemi-orchestration11.29B6.47 (24)6.157.36
Qwen2.5-Omni-7B+Nitro-Tsemi-orchestration13.54B6.60 (22)6.377.70
Dynin-Omniany-to-any (1-pass joint generation)8.35B6.94 (15)6.286.95
HyperCLOVA X 8B Omniany-to-any (per-modality invocation)≥13.27B6.20 (26)5.576.73
Omni-Diffusionany-to-any (per-modality invocation)8.63B5.56 (28)4.605.41
AnyGPTany-to-any (per-modality invocation)10.66B4.46 (32)3.133.89

*Including encoders, tokenizers, and decoders. For reference, the integrated judge range of the 18 orchestration configurations is 7.19–9.00.

*Judge mean = the mean of the 4 text, image, speech, and integrated judge scores (the judges-only aggregation in Table 28).

Table G3. Mean judge scores by the number of characters visible according to the conditions (condition difficulty) (1–10 scale)

Number of characters on the next page0123 or more
Number of transitions32293453122
image judge: large orchestration6.616.446.346.05
image judge: small orchestration5.535.204.964.70
image judge: semi (image experts)5.004.854.664.44
image judge: semi (TTS experts)3.302.401.971.96
image judge: any-to-any4.063.693.533.51
image judge: all5.415.104.924.73
integrated judge: all7.647.717.617.59

Between strata, the Spearman correlation of the rankings of the 32 configurations based on the image judge is 0.957–0.994.

AuthorsOpenReview ↗

General Response (2/3)

Table G4. Expanded human evaluation: agreement between the ranking of the 32 configurations by human scores and the judge ranking (Spearman ρ)

textimagespeechintegratedcomposite
Ratings1,0241,0241,0241,0244,096
ρ (vs. judge ranking on 900 transitions)0.9650.9760.8430.9790.988
Kendall τ0.8700.8800.6510.9050.934
Leave-one-transition-out ρ range0.947–0.9750.973–0.9840.826–0.8630.971–0.9810.983–0.989
Top-10 overlap9/109/107/108–9/10 (tie)9/10
Lowest-ranked configurationSame (AnyGPT)Same bottom 3Same (AnyGPT)Same (AnyGPT)Same (AnyGPT)

The human ranking on the composite (mean of the four tracks; on the judge side, the same quantity as the 'judge mean' in Table G2) also correlates with Total at ρ=0.979 (τ=0.910, top 10: 10/10), and the paradigm ordering (large orchestration > small orchestration > semi-orchestration > any-to-any) is the same as those of the judge composite and Total. On the basis of the 18 speech systems formed by grouping configurations that produce the same speech, the speech ρ is 0.899.

Condition stratumcomposite ρ (within-stratum human ranking vs judge ranking on 900 transitions)
Character change: new character appears / none0.988 / 0.959
Scene change: large / small0.971 / 0.985
Emotion: positive / negative / neutral0.984 / 0.959 / 0.932
Speaking speed: normal / fast0.989 / 0.963
Pitch: normal / high0.980 / 0.954
Speaker gender: female / male0.984 / 0.973
Speech emotion: happy / neutral / sad0.980 / 0.957 / 0.917
Source: storyweaver / digitallibrary0.986 / 0.986

*The evaluated items are the same output files as in the main results, and we collected all of the planned 4,096 ratings (32 configurations × 4 tracks, namely text, image, speech, and joint, with 1,024 per track; 52 annotators in total).

*The transitions were drawn from among the valid transitions in a fixed-seed order so that all configurations are evaluated on the same inputs, and their mean judge score is on par with that of the 900 transitions (6.58 vs 6.61). The per-stratum composite ρ is 0.917–0.989 at all levels with 5 or more transitions, and 0.975/0.966 on the easier and harder halves based on judge scores.

*Stratum labels are proxies defined from the dataset conditions: new character appears = there is a character in the conditions whose name does not appear in the current narration; large scene change = the HSV color-histogram overlap between the current image and the ground-truth next image is in the bottom 1/3 among the 900 transitions; emotion = whether negative or positive words appear in the emotion expressions of the characters in the conditions (neutral or other if neither appears); speech conditions = GT speech attributes.

*In the revision, we will replace the small-scale evaluation in App. F.5 with this expanded evaluation and will also revise the related descriptions accordingly.

AuthorsOpenReview ↗

General Response (3/3)

Table G5. Cost and latency measurements (19 configurations; a single A100-80GB, batch 1, identical runtime conditions)

FamilyConfigurationMean latency (s)P95 latency (s)GPU memory max / mean (GiB)Total (rank)
small orchestrationInternVL3.5-4B+Nitro-T+Parler-large35.747.229.2 / 26.26.40 (25)
small orchestrationInternVL3.5-4B+FLUX.1-dev+Parler-large56.869.255.4 / 52.76.68 (21)
small orchestrationInternVL3.5-4B+FLUX.1-Kontext+Parler-large61.673.255.4 / 52.76.93 (16)
small orchestrationQwen3.5-4B+Nitro-T+VoxCPM227.232.129.4 / 21.76.57 (23)
small orchestrationQwen3.5-4B+FLUX.1-dev+VoxCPM248.755.655.5 / 48.46.84 (19)
small orchestrationQwen3.5-4B+FLUX.1-Kontext+VoxCPM254.560.355.5 / 48.47.00 (14)
semi (image experts)Qwen2.5-Omni-7B+Nitro-T18.927.043.5 / 30.16.60 (22)
semi (image experts)Qwen2.5-Omni-7B+FLUX.1-dev41.153.269.7 / 56.76.89 (18)
semi (image experts)Qwen2.5-Omni-7B+FLUX.1-Kontext44.553.569.7 / 56.77.17 (13)
semi (image experts)EMOVA-7B+Nitro-T23.828.753.5 / 39.86.47 (24)
semi (image experts)EMOVA-7B+FLUX.1-dev40.348.367.2 / 65.06.75 (20)
semi (image experts)EMOVA-7B+FLUX.1-Kontext48.458.967.2 / 65.06.90 (17)
semi (TTS experts)MMaDA+Parler-large66.067.330.8 / 30.75.37 (30)
semi (TTS experts)MMaDA+VoxCPM264.466.427.2 / 27.15.32 (31)
semi (TTS experts)Emu3-9B+Parler-large691.5712.465.3 / 65.35.56 (27)
semi (TTS experts)Emu3-9B+VoxCPM2663.8673.666.6 / 63.05.45 (29)
any-to-any (per-modality invocation)AnyGPT20.427.322.4 / 22.24.46 (32)
any-to-any (per-modality invocation)Omni-Diffusion71.382.819.7 / 19.45.56 (28)
any-to-any (1-pass joint generation)Dynin-Omni62.864.217.9 / 17.96.94 (15)

*All configurations: a single NVIDIA A100-SXM4-80GB per baseline, batch 1, identical runtime conditions (running each model's original PyTorch, Transformers/Diffusers implementation directly at its original precision; no inference engine such as vLLM, and no torch.compile, quantization, or CPU offload). Latency and GPU memory are values measured on transitions in which text, image, and speech were generated to completion. Latency = the time from preprocessing through decoding and file saving (excluding model loading and warm-up). GPU memory = PyTorch peak allocated for the process with all components loaded (max / mean of per-transition peaks). Total (rank) is the paper's Total (the four-decimal values in Table 14 rounded to two decimals; the same values as in Table G2) and the rank among the 32 configurations.

*The 12 large-VLM-backbone configurations and HyperCLOVA X were excluded because they could not be measured under the same conditions (the 2 API models have non-public weights; GLM-4.6V 106B, whose BF16 weights alone are about 212 GB, does not fit on a single A100-80GB without quantization or offload; and HyperCLOVA X's official serving stack, OmniServe, is vLLM-based).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.