- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies visual forgetting in autoregressive video generation when previously observed content leaves the sliding context window. MosaiChunk partitions historical KV entries into equal-sized, content-based sections and retrieves sections from different spatial regions and time points under a fixed active-cache budget. The video backbone remains frozen, while a descriptor encoder is trained by matching predictions from the same backbone with richer historical context. The authors also introduce RememBench, covering prompt-driven object reappearance and camera-driven scene revisits. Experiments on two backbones show improvements over sliding-window inference and an adapted whole-chunk retrieval baseline, with additional evaluations on WBench and WorldMark.
Critical strengths
-
The paper provides useful evidence that a frozen video generator can reuse selected portions of historical KV. The region-selection example and the cross-rollout KV transplantation experiment make this observation concrete. This suggests a practical way to improve memory without updating the generator.
-
The T2V results show a meaningful improvement under matched active-cache budgets. At the two-chunk budget, MosaiChunk improves CLIP similarity from 0.841 to 0.936 and reduces LPIPS from 0.569 to 0.500 relative to the adapted MoC baseline. The improvement is consistent across the two reported budgets, and the evaluation covers both object reappearance and scene revisits.
Critical weaknesses
-
The distinction from closely related memory methods is incomplete. SlotMemory already organizes original KV tokens through semantic routing addresses, while LayerRecall freezes the generative backbone and trains a bounded-memory router by matching predictions from a privileged long-context teacher. These overlap with two central aspects of this paper, yet neither is discussed in the submitted version. MosaiChunk differs in its equal-sized section construction and composition, but the paper needs to explain what this design adds beyond existing semantic KV memory and prediction-matching approaches. My concern is the extent of the contribution, rather than whether the implementations are identical.
-
The experiments do not yet establish a substantial benefit from section composition over a strong whole-chunk retriever. Table 13 is particularly relevant. With two chunks and pose guidance, MosaiChunk-Pose improves rotation CLIP only from 0.891 to 0.893 over WorldKV. With translation, CLIP changes from 0.897 to 0.899, while LPIPS worsens from 0.542 to 0.546. At the one-chunk rotation setting, WorldKV performs better on both metrics. These results leave the added value of section composition uncertain when retrieval is already effective. The larger gains over MoC do not resolve this issue: the adapted MoC uses pooled-key retrieval, whereas MosaiChunk has a trained descriptor encoder and learned value weights. Table 4 also changes the descriptor representation and value weighting together. A whole-chunk retriever trained with the same supervision would help determine how much of the gain actually comes from the proposed memory granularity. Without this control, I find the evidence for the central architectural contribution insufficient.
Questions for authors
-
Clarification on novelty: How does the proposed memory organization differ from SlotMemory, and how does oracle-memory self-distillation differ from LayerRecall’s prediction-matching training? Which contribution remains specific to MosaiChunk after accounting for these methods?
-
Evidence for the current architectural claim: Can the authors compare section retrieval with whole-chunk retrieval using the same learned descriptor, teacher supervision, and value weighting? This would directly test whether section composition, rather than better-trained retrieval, explains the gains.
-
Clarification on the strength of the results: How should the near-ties with WorldKV in Table 13 be interpreted? Are the small differences consistent across scenes or seeds? Paired results or uncertainty estimates would help assess whether composition provides a reliable advantage.
2 comments
Response to Reviewer otnX (2/2): WorldKV Comparison
Reply 2/2
Q3. How should the near-ties with WorldKV be interpreted across scenes and seeds?
Existing pose-based retrieval mechanisms such as WorldKV’s require an input camera trajectory and are best suited to static scenes. In dynamic scenes, objects may move or change between visits, so retrieving memory from the same camera pose may return content that no longer matches the current scene. MosaiChunk instead retrieves memory by visual content without requiring camera trajectories, providing a general memory mechanism for KV-cached autoregressive video generation.
As stated in Appendix C.3, we do not claim a consistent advantage over WorldKV. This ablation examines whether pose can provide an additional retrieval signal for MosaiChunk. MosaiChunk-Pose uses camera-pose similarity to shortlist historical chunks, then applies the same trained router to select sections within them. For this comparison, WorldKV uses its pose-based whole-chunk retrieval on our frozen backbone with matched budgets, without World Compression. We evaluated two additional noise seeds (three in total) on all 150 scenes with 180° rotation, demonstrating that MosaiChunk-Pose and WorldKV consistently achieve similar performance across seeds.
| Seed | WorldKV | MosaiChunk-Pose | |
|---|---|---|---|
| 1 | Original | 0.853 / 0.610 | 0.843 / 0.624 |
| 1 | Additional 1 | 0.869 / 0.594 | 0.861 / 0.600 |
| 1 | Additional 2 | 0.856 / 0.605 | 0.851 / 0.598 |
| 2 | Original | 0.891 / 0.578 | 0.893 / 0.577 |
| 2 | Additional 1 | 0.905 / 0.537 | 0.902 / 0.545 |
| 2 | Additional 2 | 0.895 / 0.554 | 0.894 / 0.566 |
Cells report median CLIP ↑ / LPIPS ↓.
Response to Reviewer otnX (1/2): Novelty and Retrieval Ablations
Reply 1/2
We thank the reviewer for their thoughtful feedback. We appreciate their recognition of historical KV reuse and the T2V improvements under matched memory budgets. Below, we clarify the differences from related work and provide additional whole-chunk and WorldKV comparisons. We will incorporate these results and clarifications in a revised version.
Q1. How does MosaiChunk differ from SlotMemory and LayerRecall?
We thank the reviewer for pointing out these two closely related works. Below, we clarify the fundamental differences in memory organization and the distillation setup. Some components are complementary and could be combined with our approach. We also note that LayerRecall appeared as a preprint shortly before our submission; we therefore regard it as concurrent work. We will discuss both works in a revised version.
SlotMemory uses learned, object-centric KV slots and combines text and visual cues for retrieval. MosaiChunk instead uses equal-size sections without an object prior and composes them across historical chunks through visually cued retrieval. This allows us to retrieve both objects and backgrounds without relying on text cues, supporting revisits that are not explicitly signaled in the prompt.
LayerRecall matches a longer-context teacher with an independent rollout and retrieves whole chunks at selected layers. Our teacher and student differ only in far memory to train section composition; LayerRecall's layer selection is orthogonal and could be combined with our approach.
MosaiChunk offers two key contributions:
Q2. Does section composition help beyond learning a stronger retrieval descriptor?
Following the reviewer’s suggestion, we retrain whole-chunk routers for both I2V and T2V and find that they outperform MoC but fall short of MosaiChunk, as shown in the tables below. These results support the benefits of both learned descriptors and section composition, which MosaiChunk combines. Descriptor training encourages retrieval of relevant historical content over recent content, while section composition allows useful information from several historical chunks to share the same budget. Visual comparisons agree with the quantitative results. We will include these visualizations, including outputs from the retrained whole-chunk checkpoints, in a revised version.
We keep the descriptor architecture, teacher supervision, value weighting, and optimization settings unchanged. During training and inference, the router retrieves complete chunks ranked by their maximum section similarity to the query. Both I2V and T2V evaluations use a one-chunk far-memory budget. Due to time constraints, the I2V comparison focuses on 180° rotation.
I2V (180° rotation)
T2V
Additionally, MosaiChunk supports fractional-chunk budgets and does not require an integer number of chunks. MosaiChunk achieves comparable performance to retrained whole-chunk retrieval with half a chunk less memory (1.5 vs. 2 chunks).