acceptodds
Under review as a conference paper at ICLR 2027

Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation

Abstract

Unified multimodal models (UMMs) provide an interface for interleaved text-image generation, where generated visual states can directly inform subsequent reasoning and synthesis. However, existing systems are still primarily demonstrated on short trajectories or simplified visual content. We develop a large UMM natively trained on interleaved multimodal data, with strong capabilities in image synthesis, text rendering, complex composition, and multimodal planning, and study how to extend these native capabilities to substantially longer autoregressive trajectories. The main challenge is that generated images continuously accumulate in context: irrelevant or imperfect visual states interfere with future synthesis, while previously established characters and scenes must still be recovered after long intervals. We introduce a visual-memory framework that maintains persistent visual references, selectively retrieves scene-relevant historical states while preserving the complete semantic history, and uses few-step self-rollout to adapt the model to imperfect generated histories. We further introduce LongMIG-Bench for evaluating long-horizon text-to-multi-image and image-to-multi-image generation. Our model achieves high-fidelity long-horizon interleaved generation with strong character and scene consistency, demonstrating how visual memory can scale native interleaved generation to substantially longer trajectories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Models, Interleaved Generation, Image Generation, Visual Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dVreOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a practical system recipe for scaling native interleaved text-image generation in unified multimodal models (UMMs) to long horizons (10 to 50+ images). Built on a 76B Mixture-of-Transformers (MoT) backbone (an understanding stream processing text and 1K ViT semantic tokens per image, and a flow-matching generation stream processing 4K VAE latent tokens per image), the authors combine three components: (1) Persistent Visual Memory, which generates 1:1 character reference images inside the <think> block and keeps them in context as identity anchors, (2) Scene-Aware Visual Memory Retrieval, which keeps the full history of lightweight ViT semantic tokens for global reasoning while pruning dense VAE tokens so image synthesis only attends to character anchors, recent frames, and same-scene frames, and (3) Few-Step Self-Rollout, a 500-step offline post-training stage that replaces clean historical conditioning images with model-generated ones to reduce exposure bias. Evaluated on the proposed 100-story LongMIG-Bench, the method shows strong visual consistency and lower long-horizon generation latency.

Critical strengths
  1. KV-cache pruning across the dual streams of the MoT is insightful. Decoupling historical KV retention between the understanding and generation streams—keeping all lightweight ViT semantic tokens (1K/image) for the understanding pathway while selectively pruning dense VAE tokens (4K/image) for the generation pathway—looks novel at first glance and makes total sense upon closer inspection. It hits a nice sweet spot between preserving complete global story memory and avoiding visual interference and attention bloat during image synthesis, cutting generation context growth by 3.3x and reducing 32-image prefill latency from 69.0s to 19.4s (Table 6, Figure 12).

  2. A well-engineered system with sufficient experiments and valuable implementation experience. Although most individual design choices are adaptations of existing practices, putting them together into a native 76B interleaved UMM and scaling it to 20-50+ image trajectories brings valuable implementation experience to the community. The experimental validation is thorough, including controlled same-backbone comparisons (UniLongGen* in Table 1 and GT-FT vs. Self-Rollout in Table 2 and Figure 5) that clearly verify the effectiveness of each stage.

Critical weaknesses
  1. The method does not deliver many new algorithmic insights outside of dual-stream KV pruning. Aside from the ViT/VAE KV-cache split, most components are well-known and widely adopted in related works: generating character reference sheets upfront is standard in commercial and agentic workflows (e.g., Midjourney character references and story agents), historical context curation builds directly on UniLongGen (Chen et al., 2026a), and replacing clean teacher-forced history with model rollouts closely follows Self-Forcing (Huang et al., 2025).

  2. Rule-based scene routing and omitted implementation hyperparameters. The scene-aware retrieval mechanism is not a learned memory module: it requires supervised discrete scene labels during training (Equation 5), relies on parsing a discrete scene ID from generated text at test time (Appendix H.2), and applies a hand-crafted heuristic rule (Equation 7). Furthermore, key hyperparameters for this rule—specifically the recent window size KrK_r, early same-scene budget NN, and recent same-scene budget MM in Equation 7, as well as the rollout substitution probability in Equation 10—are omitted from both the main text and the appendix.

Questions for authors
  1. Missing hyperparameters for reproducibility: Since a major contribution of this work is its practical implementation recipe, could you report the exact values of KrK_r, NN, and MM in Equation 7, as well as the substitution probability and selection rule for SrollS_{\text{roll}} in Equation 10?
  2. Robustness of text-parsed scene routing: Because scene retrieval at inference time depends on parsing a discrete scene ID emitted in the text reasoning trace (Appendix H.2), what happens when a story involves continuous location transitions or when the model fails to emit a valid scene ID?
  3. Human evaluation vs. GPT-6 judge: Since the main quality and memory metrics in Table 1 rely on GPT-6 judge prompts, did you run any small-scale human preference check to confirm agreement with the automated judge?
2 comments
AuthorsOpenReview ↗

Response to Reviewer dVre

We thank the reviewer for the thorough and constructive assessment, and for recognizing the dual-stream KV design. We address contribution positioning, retrieval and rollout hyperparameters, scene-routing robustness, and human evaluation below.

1. Contribution relative to prior work

The contribution is stream-specific visual-memory retention in a native interleaved UMM: the understanding stream keeps the full text and ViT history, while the generation stream sees only a few scene-relevant VAE states. The split, not a new retrieval rule, is the point: the two streams retain different amounts of visual history, so story memory stays complete while synthesis attention stays bounded.

We agree that references and rollout adaptation build on existing practice and cite UniLongGen and Self-Forcing. We differ from UniLongGen in three respects. (i) What is filtered: it drops less relevant historical states as whole units, while we keep the semantic trajectory and expose only dense generation states. (ii) When: it is training-free, curating only at inference, while our scene state is supervised during training (Eq. 5) and Eq. 7 is applied to teacher-forced trajectories, so training and inference share the same visual-context distribution. (iii) Cost: it runs an extra curation step at inference, while our selection uses only the emitted scene ID and a fixed budget (K_r = 2, N = 1, M = 1), a deterministic lookup with no extra model.

References are generated in the model's own <think> trace (Eq. 4) and persist as memory; removing them drops LR Scene from 3.10 to 2.03 (Table 2), so they also support scene memory. Rollout adaptation is offline, keeps ground truth, and takes about 500 steps. On the same backbone, references and scene-aware retrieval raise Scene from 2.61 to 3.35 and Visual Memory from 2.60 to 3.77 (Table 1); at 32 images, time per image falls from 102.9 to 54.9 s and last-image prefill from 69.0 to 19.4 s. We will state this positioning and the training-time routing explicitly in Section 3.3.

2. Retrieval and rollout hyperparameters

The default configuration is:

ParameterValue
K_r2
N1
M1
Rollout probability0.8
Ground-truth prefix2

Scene indices are zero-based, the recent window is 0 < k − j ≤ K_r, and training and inference use the same rule. Deduplication gives at most four non-reference VAE images plus all references.

Rollout selection uses one Bernoulli draw per story; if enabled, references and the first two story frames keep ground truth, eligible later positions use precomputed rollout images, and the prefix is capped at 13 image-list slots including references. Both VAE and ViT representations are recomputed; text and diffusion targets stay ground truth; understanding/ViT parameters are frozen. Rollouts use 30 Euler steps and CFG 3.5; self-rollout uses learning rate 3×10⁻⁵ and batch size 32. We will add the full table to the appendix.

3. Scene routing and continuous transitions

Scene state is predicted by the understanding stream in the text trace, a learned behavior; the controller mapping it to the retrieval set is deterministic. If the tag cannot be parsed, retrieval falls back to persistent references plus the two most recent story frames, keeping the full text/ViT history. On our long-horizon set (up to 24 images per story, 921 scene slots), scene state matches the annotation on 82.6% of slots, and the model reuses its own label on 91.0% of 804 revisits. On the 24-image LONG+EXT test used for Table 2, predicted and ground-truth routing perform comparably in Scene.

For continuous location transitions, the recent-frame window does not depend on scene segmentation, so short-range continuity is preserved even when the segmentation is ambiguous, and the global semantic history remains available for planning. With 30% noisy scenes, Scene falls from 3.26 to 2.81, still above Recent-2 (2.58), so label errors degrade retrieval smoothly rather than breaking generation. We will document the controller and fallback in Appendix H.2.

4. Human evaluation versus the GPT-6 judge

To test the judge directly, we ran a blinded human study: 7 methods, 30 stories, 1,512 images, 11 evaluators, 1,155 valid ratings on a 1–5 scale. Ours ranks first on cross-image consistency (3.63), aesthetics (3.49), and text rendering (4.14), matching its first-place GPT-6 Visual Memory (3.77) and Aesthetic (3.75). Qwen-Image is the only competitive system on generic image quality (3.32 vs 3.47 human; 3.83 vs 3.75 GPT-6 VF). Repeated GPT-6 scoring has a within-item SD of 0.087 (Scene) and 0.153 (Visual Fidelity), far below the memory gap (3.77 vs 2.60, Table 1). We will include the full protocol in the appendix.

The authors have resolved my concerns. Given the score is already highest, I will maintain my score :).

Reviewer_cpGtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a unified model for long interleaved text-image generation, which combines visual references, scene-based retrieval of past image latents, and adaptation using self-rollout history. It also introduces LongMIG bench and reports ablation ablations and longer generation examples.

Critical strengths

Separating semantic history from visual memory makes sense and also reduces the context. The ablation provide useful insights. The method halves generation time in the 32-image setting.

Critical weaknesses

The quality evaluation lacks human validation and does not clearly account for missing requested content. Some implementation details are also unclear, especially the retrieval budgets and whether self-rollout updates the semantic history.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer cpGt

We thank the reviewer for the careful reading and for recognizing the motivation for separating semantic history from visual memory. We address the concerns about human validation, missing requested content, retrieval budgets, and self-rollout below.

1. Human evaluation

To address the absence of human judgment, we conducted an additional blinded human study: 7 methods, 30 stories, and 1,512 generated images. Eleven evaluators each assessed 15 stories, yielding 1,155 valid ratings on a 1–5 scale for cross-image consistency, visual quality, aesthetics, and text rendering.

MethodConsistencyVisual qualityAestheticsText renderingAverage
Qwen-Image3.463.323.133.393.33
HiDream1.962.822.732.702.55
UniSpace1.832.142.241.501.93
BAGEL2.441.851.711.451.86
SenseNova-U1.53.042.892.813.803.14
Emu3.52.192.452.422.872.48
Ours3.633.473.494.143.68

Ours is highest on all four dimensions. Qwen-Image is the strongest baseline on consistency (3.46), and SenseNova-U1.5 on text rendering (3.80), 0.34 below ours. The protocol and per-method scores will be added to the appendix.

2. Missing requested content

We additionally report Narrative Coverage, which measures how much of the requested story content appears in the images and does not depend on frame count. For each story, the GPT-6 judge receives the full specification (characters, scenes, ordered shots and events) and all generated images, and returns one 1–5 score: 5 = the complete arc and major events are recognizable; 3 = major beats missing; 1 = the story cannot be recovered; 2 and 4 are intermediate. Frame count is neither rewarded nor penalized, and image k need not match shot k, since native interleaved models decompose shots freely; skipping difficult shots lowers coverage because those events are not depicted.

MethodNarrative Coverage
StoryDiffusion1.51
Qwen-Image3.70
UniSpace2.76
HiDream2.79
Lance-3B2.18
Show-o2-7B3.10
BAGEL-7B1.26
SenseNova-U1.52.31
ILLUME-X1.33
Emu3.53.11
SEED-Story1.00
UniLongGen*2.95
Ours3.45

With the same backbone, coverage rises from 2.95 for UniLongGen* to 3.45 for ours (+0.50); across the 13 methods ours ranks second, behind only Qwen-Image (3.70), and first among native interleaved systems. Replacing the specification with an unrelated one lowers coverage from 2.22 to 1.00 across 35 control items, confirming that the judge scores against the specification.

3. Retrieval budgets

Both training and inference use K_r = 2, N = 1, M = 1:

MemoryBudgetRule
Recent frames2Two story images before the current step
Earliest same-scene frame1First frame of the current scene
Latest same-scene frame1Most recent frame of the current scene
Persistent referencesallRetained separately

The three sets are deduplicated, so each step sees VAE states of at most four non-reference historical images plus references; all text and ViT semantic tokens remain available. A frame selected by multiple rules is counted once, and fewer images are visible early in a scene. Recent-4 is the recency-only ablation (K_r = 4, N = 0, M = 0). We will specify the counting convention next to the retrieval equation.

4. Self-rollout and semantic history

Yes. For each substituted historical image, both the clean VAE representation and the ViT semantic representation are recomputed from the same generated image. Offline rollout generates images autoregressively; the generated image at a substituted position goes through both VAE and ViT, feeding the generation stream's visual condition and the understanding stream's semantic history.

ComponentTreatment in self-rollout
Clean VAE at a substituted positionRe-encoded from the generated image
ViT tokens at that positionRe-encoded from the same image
Historical text planningGround truth
References and first two story imagesGround truth
Current supervision targetGround truth

Substitution is enabled per story with p = 0.8; references and the first two non-reference story images keep ground truth, eligible later positions use rollout images, and positions beyond the offline prefix keep ground truth. The remaining 20% of stories use ground truth. Understanding and ViT parameters are frozen, so their inputs change while the modules are fixed; generation parameters are updated. We will make this procedure explicit in the method section.

Reviewer_dCPkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a practical problem: scaling the unified multimodal models to long-horizon interleaved generation. To achieve this goal, a visual-memory framework with character references, historical visual state retrieval, and self-rollout post-training on histories is proposed. Besides, this paper proposes LongMIG-Bench to validate the effectiveness of the proposed method.

Critical strengths
  1. This paper is well-organized and easy to follow.
  2. The studied problem is practical and useful to address the existing limitations of UMMs on long-horizon interleaved generation.
  3. The proposed benchmark comprehensively includes vision domains and scenes. The additional experiments on latency is effective and demonstrate the proposed method is useful to reduce context when trajectories become longer.
Critical weaknesses
  1. One of my main concerns is that the evaluation heavily depends on GPT-6 scores without human judgment. It is unclear whether these evaluations are reliable, especially for those results with only small differences.
  2. Although the method selects fewer image histories to reduce context, it still keeps the full semantic visual history. It is necessary to report the KV-cache usage and GPU memory when generating long sequences with full semantic history.
  3. More implementation details, such as the number of historical images that are retrieved and the initialization of the backbone, should be included.
  4. Does this framework generalize to long-horizon scenarios beyond the training data?
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer dCPk

We thank the reviewer for the positive assessment of our benchmark and efficiency analysis, and for the concrete suggestions on human validation, memory measurement, implementation details, and length generalization. We address the four concerns below.

1. Human evaluation

To address the absence of human judgment, we conducted an additional blinded human study: 7 methods, 30 stories, and 1,512 generated images. Eleven evaluators each assessed 15 stories, yielding 1,155 valid ratings on a 1–5 scale for cross-image consistency, visual quality, aesthetics, and text rendering. The average is the equally weighted mean of the four dimension-level scores.

MethodConsistencyVisual qualityAestheticsText renderingAverage
Qwen-Image3.463.323.133.39 (19)3.33
HiDream1.962.822.732.70 (15)2.55
UniSpace1.832.142.241.50 (8)1.93
BAGEL2.441.851.711.45 (24)1.86
SenseNova-U1.53.042.892.813.80 (24)3.14
Emu3.52.192.452.422.87 (17)2.48
Ours3.633.473.494.14 (24)3.68

Ours is highest on all four dimensions. Against Qwen-Image we improve consistency, visual quality, and aesthetics by 0.17, 0.15, and 0.36 points, respectively; on text rendering we reach 4.14, 0.34 above SenseNova-U1.5 (3.80). Text-rendering scores cover stories with text (valid counts in parentheses). The human ranking agrees with the automated results on the memory-related axes and does not depend on the GPT-6 judge. Full per-method results and the protocol will be added to the appendix.

2. KV-cache and GPU memory

Our method keeps the full ViT semantic history, but only the VAE states of selected images enter synthesis; retrieved images are re-encoded from CPU storage. This preserves the semantic history of characters, events, and scenes while controlling the number of dense visual representations through the retrieval budget. Both arms use identical weights and hardware: AR and DiT on separate 8-GPU nodes (TP=8, 16 GPUs, 95 GiB each), 30 denoising steps, CFG 3.5.

The generation stream has 48 layers with GQA (4 KV groups, head dim 128); in bf16 the theoretical KV payload is 2×48×4×128×2 = 98,304 B = 96 KiB per context token. The table reports the image-synthesis context and KV payload from a single-story trace; indices are zero-based.

Image indexFull History contextOurs contextFull KV (GiB)Ours KV (GiB)
844,41633,7604.073.09
1684,12748,4987.704.44
24124,23460,05711.375.50
32164,64571,91615.076.58

At 32 images the context is 92,729 tokens (56.3%) smaller, so KV falls from 15.07 to 6.58 GiB; growth is 5,028 vs 1,546 tokens per image (about 3.3×). These counts include the full text and ViT history.

HorizonFull History (MB)Ours (MB)Reduction
8 images17,60617,1532.6%
16 images20,90617,93314.2%
24 images24,41318,85922.8%
32 images28,53820,64827.6%

At 32 images, measured DiT peak reserved memory per GPU is 7,890 MB lower; for the two paired trajectories reaching 32 images, growth from 8 to 32 images falls from 10,890 to 3,350 MB (about 3.3×).

3. Retrieval budget and initialization

Retrieval uses K_r = 2, N = 1, and M = 1: the two most recent frames, the earliest frame of the current scene, and its latest frame. Deduplication leaves at most four non-reference historical VAE images plus persistent references. Long-horizon post-training starts from our MoT checkpoint after pretraining and interleaved SFT; the understanding stream is initialized from a pretrained 30B/3B MoE language model, and the generation stream loads its own attention and feed-forward parameters. We will give the full initialization and training stages in the appendix.

4. Beyond the training horizon

Training stories contain 2–24 panels, while the paper shows trajectories beyond 50 images (Fig. 4), over twice the training maximum. This behavior follows from the design: retrieval selects visual memory by scene and position, and its budget does not depend on a predetermined total length. As the trajectory grows, synthesis always sees at most four non-reference images plus references, while the full semantic history keeps characters, events, and scenes. The 32-image results above (27.6% lower peak memory) also reduce the cost of longer trajectories. We will clarify the training-length range and include these extended-generation results in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.