acceptodds
Withdrawn from ICLR 2027

IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Humans routinely consume multimodal content, where textual and visual information is naturally interleaved. Although Unified Multimodal Models (UMMs) can jointly generate text and images, multimodal interleaved generation remains largely underexplored. Progress in this area is hindered by two key limitations: (1) the lack of well-defined interleaved generation tasks and corresponding training data, and (2) the absence of fine-grained evaluation protocols that can reliably diagnose model performance across different capability dimensions. To this end, we present **IntReal**, comprising two complementary components: **IntReal-100K**, a real-world dataset of multimodal interleaved samples spanning four major categories, and the **IntReal Evaluation Suite**, which evaluates model performance across seven complementary dimensions using specialized expert models, enabling coarse-to-fine diagnosis. Extensive experiments validate our evaluation suite through low inter-metric redundancy (max pairwise absolute Spearman's ), high agreement with human judgments (83.3% accuracy), and strong consistency across different expert evaluators (Spearman's ), supporting the complementarity and reliability of our metrics. By benchmarking four families of UMMs, we show that IntReal Evaluation Suite reveals distinct strengths and weaknesses across models. Post-training experiments demonstrate the utility of IntReal-100K for improving interleaved generation and further show that training unimodal generation can also benefit interleaved generation. These findings suggest that interleaved generation can benefit from modality-specific optimization, rather than requiring joint text–image training at every stage. *We will release IntReal-100K, the fine-tuned models, and the evaluation code.*

Primary area:
datasets and benchmarks
Keywords:
Multimodal Generation, Unified Multimodal Models, Dataset and Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_68goOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces IntReal, a dataset and benchmark for real-world interleaved text-image generation. It includes 100K high-quality interleaved samples and seven fine-grained evaluation metrics to assess and improve unified multimodal models.

Critical strengths
  • An important but underexplored task. Interleaved text-image generation is a promising direction for unified multimodal models, but dedicated datasets and benchmarks are still limited.

  • I appreciate the substantial engineering effort. I really value dataset-oriented research and the effort the authors put into building this resource.

Critical weaknesses
  • One of the main contributions is IntReal-100K, but it's unclear how much the proposed filtering steps actually help interleaved generation. How does each filter affect data quality and distribution? Do these improvements translate into better downstream performance? I think some ablation studies are needed to justify these design choices.

  • Text richness and logical coherence are quite subjective and hard to evaluate. I'm not fully convinced that using an LLM judge with generic prompts can reliably measure these properties. While the human agreement study is helpful, it doesn't necessarily show that the metrics capture what they're supposed to measure. I would prefer sample-specific, verifiable rubrics, similar to what the authors use for Context Fidelity.

  • I'm not sure Image Aesthetic and Image Diversity accurately reflect the actual image generation capabilities of different models. For example, Emu3.5 and NanoBanana2 have very similar aesthetic scores in Table 3, but is their real-world image generation quality really that close? Also, DINO-based diversity may be misleading, since interleaved generation often requires visual consistency across images. I'd like to see more direct human preference evaluations to validate these metrics.

  • The authors observe that SFT hurts some image generation metrics and then use RL to recover performance. But I think the more interesting question is why SFT on IntReal-100K hurts performance in the first place. Is it due to the data distribution, training setup, or the proposed multimodal optimization conflicts? The paper doesn't investigate this enough before moving to RL. Even if RL improves the scores, that doesn't mean the degradation is unavoidable or that RL is the best solution. I'd like to see a deeper analysis of what causes the SFT degradation, rather than just showing how much RL can recover.

Questions for authors

See weakness

Reviewer_EcVyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces a dataset for interleaved image–text generation, together with a companion evaluation suite. It contains 100,000 sequences and 538,452 images from seven sources, grouped into four categories: planning, knowledge, tutorial, and story. A held-out evaluation set of 600 samples assesses generation along seven dimensions under two settings: generating a full interleaved image–text response, and generating only the next image given a fixed real preceding context. Using this dataset, the authors evaluate several unified multimodal models, and also fine-tune an open-source UMM themselves with SFT and with SFT+RL to test whether the data is effective. They report some useful findings: SFT improves text performance but degrades some image capabilities, while RL recovers part of that loss.

Critical strengths
  1. The proposed dataset is genuinely valuable. Existing open-source image–text datasets mostly serve multimodal understanding rather than better generation. The filtering criteria, sources, and related details in the appendix make it easier to understand what this data actually is.

  2. Human comparison and cross-checks across multiple LLMs improve the reliability of the metrics. The authors do not simply assume that LLM evaluation is reliable. In Section 4.4 they report human pairwise comparisons on five metrics, with an average agreement of 83.3%. They also use Qwen3.6 to cross-check the Gemma4 scores on the benchmark, showing that the rankings do not depend on a single judge model.

Critical weaknesses
  1. The seven metrics miss a critical part of interleaved image–text generation. They cover text, image appearance, repetition, and image count, plus Context Fidelity and Subject Consistency under a fixed real context. They do not check, over the full generation process, whether images correspond to the generated text, nor whether the model keeps subject identity consistent across the entire sequence it produces itself. A response can have plausible text and rich, visually appealing images while the image–text pairing is wrong, and the overall evaluation has no explicit penalty for that mismatch. For a benchmark whose main target is interleaved image–text generation, I consider this a substantive coverage gap.

  2. The claim that IntReal-100K improves interleaved generation is not well supported. First, the paper does not compare whether SFT on ordinary image–text datasets yields the same gains, and it reports no results on any external benchmark (such as OpenING or ISG-Bench). Second, the experiment rests on only one base-versus-SFT comparison (U1-8B-MoT-Interleaved). With Gemma4 as judge, the mean moves from 71.88 to 72.31 (+0.60%); with Qwen3.6 it moves from 71.58 to 71.47 (−0.15%, Table 7). At the same time, image aesthetic drops by 5.65% and context fidelity by 4.89%. Third, for U1, RL raises the average from 72.31 to 73.81, but Image Aesthetic and Context Fidelity remain below the original model, which if anything further suggests that the current data has a clear negative effect on interleaved generation. Finally, contribution 3 is circular under the present experiments, because the signal optimized by RL is also used in the benchmark. A higher evaluation score shows that the reward can be optimized, but it is not enough to show that broader interleaved image–text generation ability has improved.

Questions for authors
  1. Broader experimental validation. Fine-tune U1-8B-MoT for the same number of steps on CoMM or InterSyn, and separately on the unfiltered raw pool, then report results on IntReal plus one external benchmark. If IntReal-100K clearly wins on both, I would raise my score.
  2. Coverage of the evaluation dimensions. In fully free-form generated responses, can the existing metrics measure image–text mismatch or long-range consistency of subject identity? If you can provide a few such failure cases together with their scores and thereby show that the metric set is complete, I would raise my score.
  3. Some details need clarification. Why does RL for U1-8B-MoT use only RALI? Could you report image aesthetic with another IQA model, and run a small-scale human preference test of base, SFT, and SFT+RL?
  4. Soundness of the evaluation set. When the 600 evaluation samples were selected, were they also isolated by source page or by video episode? How is overlap with the training set kept as low as possible?
Reviewer_dEs9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces IntReal to evaluate multimodal interleaved generation in Unified Multimodal Models (UMMs). Specifically, it includes IntReal-100K, a real-world dataset containing 100K multimodal interleaved samples spanning four major categories, and the IntReal Evaluation Suite, which assesses model performance across seven dimensions. The suite evaluates interleaved generation in a coarse-to-fine manner across two settings: holistic evaluation (generating complete interleaved sequences from scratch) and fine-grained evaluation (generating the next image conditioned on fixed ground-truth interleaved contexts). The authors evaluate four UMM families and investigate supervised fine-tuning (SFT) and image-focused reinforcement learning (RL) using the proposed dataset. They find that current UMMs show clear trade-offs between textual and visual capabilities, and applying RL to next-image generation mitigates the visual degradation after SFT and improves overall performance on IntReal.

Critical strengths
  1. The paper proposes a practical data curation pipeline combining representation-based filtering, VLM-based refinement, and LLM-based annotation to distill noisy web sources into a clean, structured interleaved dataset. This provides a useful resource for training and studying UMMs on multi-step interleaved generation.

  2. The benchmark introduces seven complementary metrics including textual quality, visual aesthetics and diversity, instruction following, and cross-modal consistency, providing a thorough and fine-grained diagnosis of interleaved generation capabilities in UMMs.

Critical weaknesses
  1. Similar categorized interleaved benchmarks already exist, and it is unclear whether the proposed four-category taxonomy is well-justified or overclaimed. The boundaries between categories are blurry and overlapping (in Figure 1, the Plan and Knowledge examples both read like knowledge), while each category is tightly bound to narrow sources (Xiachufang for Tutorial and three cartoons for Story).

  2. The experiment is too limited, covering only four model families and two post-training setups, and the results raise doubts about the actual quality of IntReal-100K. Notably, the evaluation set is a direct subset of IntReal-100K sharing the exact same data sources and curation pipeline. Even under this in-domain evaluation setting, SFT on IntReal-100K degrades four out of seven metrics on U1-8B-MoT, yielding only a +0.60% overall gain.

Questions for authors
  1. Please evaluate additional UMM families using both SFT and RL, and compare with their base models to better demonstrate the utility and generalizability of IntReal-100K.

  2. How are the four categories, particularly Planning and Knowledge, distinguished beyond their data sources? Please provide concrete examples and evidence that the category-level differences reflect diverse task capabilities rather than differences in data sources.

  3. Please evaluate the OOD generalization capabilities of models post-trained on IntReal-100K using external datasets or unseen domains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.