- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces IntReal, a dataset and benchmark for real-world interleaved text-image generation. It includes 100K high-quality interleaved samples and seven fine-grained evaluation metrics to assess and improve unified multimodal models.
Critical strengths
-
An important but underexplored task. Interleaved text-image generation is a promising direction for unified multimodal models, but dedicated datasets and benchmarks are still limited.
-
I appreciate the substantial engineering effort. I really value dataset-oriented research and the effort the authors put into building this resource.
Critical weaknesses
-
One of the main contributions is IntReal-100K, but it's unclear how much the proposed filtering steps actually help interleaved generation. How does each filter affect data quality and distribution? Do these improvements translate into better downstream performance? I think some ablation studies are needed to justify these design choices.
-
Text richness and logical coherence are quite subjective and hard to evaluate. I'm not fully convinced that using an LLM judge with generic prompts can reliably measure these properties. While the human agreement study is helpful, it doesn't necessarily show that the metrics capture what they're supposed to measure. I would prefer sample-specific, verifiable rubrics, similar to what the authors use for Context Fidelity.
-
I'm not sure Image Aesthetic and Image Diversity accurately reflect the actual image generation capabilities of different models. For example, Emu3.5 and NanoBanana2 have very similar aesthetic scores in Table 3, but is their real-world image generation quality really that close? Also, DINO-based diversity may be misleading, since interleaved generation often requires visual consistency across images. I'd like to see more direct human preference evaluations to validate these metrics.
-
The authors observe that SFT hurts some image generation metrics and then use RL to recover performance. But I think the more interesting question is why SFT on IntReal-100K hurts performance in the first place. Is it due to the data distribution, training setup, or the proposed multimodal optimization conflicts? The paper doesn't investigate this enough before moving to RL. Even if RL improves the scores, that doesn't mean the degradation is unavoidable or that RL is the best solution. I'd like to see a deeper analysis of what causes the SFT degradation, rather than just showing how much RL can recover.
Questions for authors
See weakness