- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a practical system recipe for scaling native interleaved text-image generation in unified multimodal models (UMMs) to long horizons (10 to 50+ images). Built on a 76B Mixture-of-Transformers (MoT) backbone (an understanding stream processing text and 1K ViT semantic tokens per image, and a flow-matching generation stream processing 4K VAE latent tokens per image), the authors combine three components: (1) Persistent Visual Memory, which generates 1:1 character reference images inside the <think> block and keeps them in context as identity anchors, (2) Scene-Aware Visual Memory Retrieval, which keeps the full history of lightweight ViT semantic tokens for global reasoning while pruning dense VAE tokens so image synthesis only attends to character anchors, recent frames, and same-scene frames, and (3) Few-Step Self-Rollout, a 500-step offline post-training stage that replaces clean historical conditioning images with model-generated ones to reduce exposure bias. Evaluated on the proposed 100-story LongMIG-Bench, the method shows strong visual consistency and lower long-horizon generation latency.
Critical strengths
-
KV-cache pruning across the dual streams of the MoT is insightful. Decoupling historical KV retention between the understanding and generation streams—keeping all lightweight ViT semantic tokens (1K/image) for the understanding pathway while selectively pruning dense VAE tokens (4K/image) for the generation pathway—looks novel at first glance and makes total sense upon closer inspection. It hits a nice sweet spot between preserving complete global story memory and avoiding visual interference and attention bloat during image synthesis, cutting generation context growth by 3.3x and reducing 32-image prefill latency from 69.0s to 19.4s (Table 6, Figure 12).
-
A well-engineered system with sufficient experiments and valuable implementation experience. Although most individual design choices are adaptations of existing practices, putting them together into a native 76B interleaved UMM and scaling it to 20-50+ image trajectories brings valuable implementation experience to the community. The experimental validation is thorough, including controlled same-backbone comparisons (
UniLongGen*in Table 1 andGT-FTvs.Self-Rolloutin Table 2 and Figure 5) that clearly verify the effectiveness of each stage.
Critical weaknesses
-
The method does not deliver many new algorithmic insights outside of dual-stream KV pruning. Aside from the ViT/VAE KV-cache split, most components are well-known and widely adopted in related works: generating character reference sheets upfront is standard in commercial and agentic workflows (e.g., Midjourney character references and story agents), historical context curation builds directly on UniLongGen (Chen et al., 2026a), and replacing clean teacher-forced history with model rollouts closely follows Self-Forcing (Huang et al., 2025).
-
Rule-based scene routing and omitted implementation hyperparameters. The scene-aware retrieval mechanism is not a learned memory module: it requires supervised discrete scene labels during training (Equation 5), relies on parsing a discrete scene ID from generated text at test time (Appendix H.2), and applies a hand-crafted heuristic rule (Equation 7). Furthermore, key hyperparameters for this rule—specifically the recent window size , early same-scene budget , and recent same-scene budget in Equation 7, as well as the rollout substitution probability in Equation 10—are omitted from both the main text and the appendix.
Questions for authors
- Missing hyperparameters for reproducibility: Since a major contribution of this work is its practical implementation recipe, could you report the exact values of , , and in Equation 7, as well as the substitution probability and selection rule for in Equation 10?
- Robustness of text-parsed scene routing: Because scene retrieval at inference time depends on parsing a discrete scene ID emitted in the text reasoning trace (Appendix H.2), what happens when a story involves continuous location transitions or when the model fails to emit a valid scene ID?
- Human evaluation vs. GPT-6 judge: Since the main quality and memory metrics in Table 1 rely on GPT-6 judge prompts, did you run any small-scale human preference check to confirm agreement with the automated judge?
2 comments
The authors have resolved my concerns. Given the score is already highest, I will maintain my score :).
Response to Reviewer dVre
We thank the reviewer for the thorough and constructive assessment, and for recognizing the dual-stream KV design. We address contribution positioning, retrieval and rollout hyperparameters, scene-routing robustness, and human evaluation below.
1. Contribution relative to prior work
The contribution is stream-specific visual-memory retention in a native interleaved UMM: the understanding stream keeps the full text and ViT history, while the generation stream sees only a few scene-relevant VAE states. The split, not a new retrieval rule, is the point: the two streams retain different amounts of visual history, so story memory stays complete while synthesis attention stays bounded.
We agree that references and rollout adaptation build on existing practice and cite UniLongGen and Self-Forcing. We differ from UniLongGen in three respects. (i) What is filtered: it drops less relevant historical states as whole units, while we keep the semantic trajectory and expose only dense generation states. (ii) When: it is training-free, curating only at inference, while our scene state is supervised during training (Eq. 5) and Eq. 7 is applied to teacher-forced trajectories, so training and inference share the same visual-context distribution. (iii) Cost: it runs an extra curation step at inference, while our selection uses only the emitted scene ID and a fixed budget (K_r = 2, N = 1, M = 1), a deterministic lookup with no extra model.
References are generated in the model's own
<think>trace (Eq. 4) and persist as memory; removing them drops LR Scene from 3.10 to 2.03 (Table 2), so they also support scene memory. Rollout adaptation is offline, keeps ground truth, and takes about 500 steps. On the same backbone, references and scene-aware retrieval raise Scene from 2.61 to 3.35 and Visual Memory from 2.60 to 3.77 (Table 1); at 32 images, time per image falls from 102.9 to 54.9 s and last-image prefill from 69.0 to 19.4 s. We will state this positioning and the training-time routing explicitly in Section 3.3.2. Retrieval and rollout hyperparameters
The default configuration is:
Scene indices are zero-based, the recent window is 0 < k − j ≤ K_r, and training and inference use the same rule. Deduplication gives at most four non-reference VAE images plus all references.
Rollout selection uses one Bernoulli draw per story; if enabled, references and the first two story frames keep ground truth, eligible later positions use precomputed rollout images, and the prefix is capped at 13 image-list slots including references. Both VAE and ViT representations are recomputed; text and diffusion targets stay ground truth; understanding/ViT parameters are frozen. Rollouts use 30 Euler steps and CFG 3.5; self-rollout uses learning rate 3×10⁻⁵ and batch size 32. We will add the full table to the appendix.
3. Scene routing and continuous transitions
Scene state is predicted by the understanding stream in the text trace, a learned behavior; the controller mapping it to the retrieval set is deterministic. If the tag cannot be parsed, retrieval falls back to persistent references plus the two most recent story frames, keeping the full text/ViT history. On our long-horizon set (up to 24 images per story, 921 scene slots), scene state matches the annotation on 82.6% of slots, and the model reuses its own label on 91.0% of 804 revisits. On the 24-image LONG+EXT test used for Table 2, predicted and ground-truth routing perform comparably in Scene.
For continuous location transitions, the recent-frame window does not depend on scene segmentation, so short-range continuity is preserved even when the segmentation is ambiguous, and the global semantic history remains available for planning. With 30% noisy scenes, Scene falls from 3.26 to 2.81, still above Recent-2 (2.58), so label errors degrade retrieval smoothly rather than breaking generation. We will document the controller and fallback in Appendix H.2.
4. Human evaluation versus the GPT-6 judge
To test the judge directly, we ran a blinded human study: 7 methods, 30 stories, 1,512 images, 11 evaluators, 1,155 valid ratings on a 1–5 scale. Ours ranks first on cross-image consistency (3.63), aesthetics (3.49), and text rendering (4.14), matching its first-place GPT-6 Visual Memory (3.77) and Aesthetic (3.75). Qwen-Image is the only competitive system on generic image quality (3.32 vs 3.47 human; 3.83 vs 3.75 GPT-6 VF). Repeated GPT-6 scoring has a within-item SD of 0.087 (Scene) and 0.153 (Visual Fidelity), far below the memory gap (3.77 vs 2.60, Table 1). We will include the full protocol in the appendix.