acceptodds
Under review as a conference paper at ICLR 2027

MosaiChunk: Compositing Spatio-Temporal Memory for Autoregressive Video Generation

Abstract

Long-horizon autoregressive video generation is limited by a finite context window. When an object or scene falls out of context, its fine-grained visual details may be lost and difficult to recover upon reappearance. To retain access to such visual details, we introduce MosaiChunk, a spatio-temporal memory mechanism that composes a mosaic of selected historical key–value (KV) entries across space and time. Our approach is motivated by the observation that a frozen video generator can directly consume such non-contiguous historical KV and recover the corresponding visual content. We therefore keep the generator fixed and learn only a lightweight router that determines which historical sections to include in the mosaic under a fixed active-memory budget. We further introduce RememBench, a benchmark of long-horizon revisits with prompt-driven text-to-video (T2V) and camera-driven image-to-video (I2V) splits. Our experiments show that MosaiChunk consistently improves revisit consistency over both sliding-window inference and whole-chunk retrieval under matched memory budgets, across both T2V and I2V settings. All code and data will be released.

Primary area:
generative models
Keywords:
autoregressive video generation, long-term memory retrieval, post-training recipe

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_otnXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies visual forgetting in autoregressive video generation when previously observed content leaves the sliding context window. MosaiChunk partitions historical KV entries into equal-sized, content-based sections and retrieves sections from different spatial regions and time points under a fixed active-cache budget. The video backbone remains frozen, while a descriptor encoder is trained by matching predictions from the same backbone with richer historical context. The authors also introduce RememBench, covering prompt-driven object reappearance and camera-driven scene revisits. Experiments on two backbones show improvements over sliding-window inference and an adapted whole-chunk retrieval baseline, with additional evaluations on WBench and WorldMark.

Critical strengths
  1. The paper provides useful evidence that a frozen video generator can reuse selected portions of historical KV. The region-selection example and the cross-rollout KV transplantation experiment make this observation concrete. This suggests a practical way to improve memory without updating the generator.

  2. The T2V results show a meaningful improvement under matched active-cache budgets. At the two-chunk budget, MosaiChunk improves CLIP similarity from 0.841 to 0.936 and reduces LPIPS from 0.569 to 0.500 relative to the adapted MoC baseline. The improvement is consistent across the two reported budgets, and the evaluation covers both object reappearance and scene revisits.

Critical weaknesses
  1. The distinction from closely related memory methods is incomplete. SlotMemory already organizes original KV tokens through semantic routing addresses, while LayerRecall freezes the generative backbone and trains a bounded-memory router by matching predictions from a privileged long-context teacher. These overlap with two central aspects of this paper, yet neither is discussed in the submitted version. MosaiChunk differs in its equal-sized section construction and composition, but the paper needs to explain what this design adds beyond existing semantic KV memory and prediction-matching approaches. My concern is the extent of the contribution, rather than whether the implementations are identical.

  2. The experiments do not yet establish a substantial benefit from section composition over a strong whole-chunk retriever. Table 13 is particularly relevant. With two chunks and pose guidance, MosaiChunk-Pose improves rotation CLIP only from 0.891 to 0.893 over WorldKV. With translation, CLIP changes from 0.897 to 0.899, while LPIPS worsens from 0.542 to 0.546. At the one-chunk rotation setting, WorldKV performs better on both metrics. These results leave the added value of section composition uncertain when retrieval is already effective. The larger gains over MoC do not resolve this issue: the adapted MoC uses pooled-key retrieval, whereas MosaiChunk has a trained descriptor encoder and learned value weights. Table 4 also changes the descriptor representation and value weighting together. A whole-chunk retriever trained with the same supervision would help determine how much of the gain actually comes from the proposed memory granularity. Without this control, I find the evidence for the central architectural contribution insufficient.

Questions for authors
  1. Clarification on novelty: How does the proposed memory organization differ from SlotMemory, and how does oracle-memory self-distillation differ from LayerRecall’s prediction-matching training? Which contribution remains specific to MosaiChunk after accounting for these methods?

  2. Evidence for the current architectural claim: Can the authors compare section retrieval with whole-chunk retrieval using the same learned descriptor, teacher supervision, and value weighting? This would directly test whether section composition, rather than better-trained retrieval, explains the gains.

  3. Clarification on the strength of the results: How should the near-ties with WorldKV in Table 13 be interpreted? Are the small differences consistent across scenes or seeds? Paired results or uncertainty estimates would help assess whether composition provides a reliable advantage.

2 comments
AuthorsOpenReview ↗

Response to Reviewer otnX (1/2): Novelty and Retrieval Ablations

Reply 1/2

We thank the reviewer for their thoughtful feedback. We appreciate their recognition of historical KV reuse and the T2V improvements under matched memory budgets. Below, we clarify the differences from related work and provide additional whole-chunk and WorldKV comparisons. We will incorporate these results and clarifications in a revised version.

Q1. How does MosaiChunk differ from SlotMemory and LayerRecall?

We thank the reviewer for pointing out these two closely related works. Below, we clarify the fundamental differences in memory organization and the distillation setup. Some components are complementary and could be combined with our approach. We also note that LayerRecall appeared as a preprint shortly before our submission; we therefore regard it as concurrent work. We will discuss both works in a revised version.

SlotMemory uses learned, object-centric KV slots and combines text and visual cues for retrieval. MosaiChunk instead uses equal-size sections without an object prior and composes them across historical chunks through visually cued retrieval. This allows us to retrieve both objects and backgrounds without relying on text cues, supporting revisits that are not explicitly signaled in the prompt.

LayerRecall matches a longer-context teacher with an independent rollout and retrieves whole chunks at selected layers. Our teacher and student differ only in far memory to train section composition; LayerRecall's layer selection is orthogonal and could be combined with our approach.

MosaiChunk offers two key contributions:

  • A novel far-memory representation: MosaiChunk assembles non-contiguous, verbatim KV sections into a virtual video chunk that the frozen generator can consume directly.
  • A general memory solution for autoregressive video generation: MosaiChunk is compatible with KV-cached autoregressive generators in both T2V and I2V. We also introduce RememBench to evaluate long-term memory in both settings.

Q2. Does section composition help beyond learning a stronger retrieval descriptor?

Following the reviewer’s suggestion, we retrain whole-chunk routers for both I2V and T2V and find that they outperform MoC but fall short of MosaiChunk, as shown in the tables below. These results support the benefits of both learned descriptors and section composition, which MosaiChunk combines. Descriptor training encourages retrieval of relevant historical content over recent content, while section composition allows useful information from several historical chunks to share the same budget. Visual comparisons agree with the quantitative results. We will include these visualizations, including outputs from the retrained whole-chunk checkpoints, in a revised version.

We keep the descriptor architecture, teacher supervision, value weighting, and optimization settings unchanged. During training and inference, the router retrieves complete chunks ranked by their maximum section similarity to the query. Both I2V and T2V evaluations use a one-chunk far-memory budget. Due to time constraints, the I2V comparison focuses on 180° rotation.

I2V (180° rotation)

MethodLearned descriptorGranularityCLIP ↑LPIPS ↓
BaseNo—0.7680.655
MoCNoWhole0.8070.645
Whole-chunk (retrained)YesWhole0.8260.629
MosaiChunkYesSection0.8400.625

T2V

MethodLearned descriptorGranularityCLIP ↑LPIPS ↓
BaseNo—0.7550.659
MoCNoWhole0.8320.594
Whole-chunk (retrained)YesWhole0.8690.575
MosaiChunkYesSection0.8990.567

Additionally, MosaiChunk supports fractional-chunk budgets and does not require an integer number of chunks. MosaiChunk achieves comparable performance to retrained whole-chunk retrieval with half a chunk less memory (1.5 vs. 2 chunks).

∥Fc∥\lVert\mathcal{F}_c\rVertMoCWhole (retrained)MosaiChunk
0.5——0.815 / 0.643
1.00.807 / 0.6450.826 / 0.6290.840 / 0.625
1.5——0.850 / 0.618
2.00.807 / 0.6420.850 / 0.6140.863 / 0.609
AuthorsOpenReview ↗

Response to Reviewer otnX (2/2): WorldKV Comparison

Reply 2/2

Q3. How should the near-ties with WorldKV be interpreted across scenes and seeds?

Existing pose-based retrieval mechanisms such as WorldKV’s require an input camera trajectory and are best suited to static scenes. In dynamic scenes, objects may move or change between visits, so retrieving memory from the same camera pose may return content that no longer matches the current scene. MosaiChunk instead retrieves memory by visual content without requiring camera trajectories, providing a general memory mechanism for KV-cached autoregressive video generation.

As stated in Appendix C.3, we do not claim a consistent advantage over WorldKV. This ablation examines whether pose can provide an additional retrieval signal for MosaiChunk. MosaiChunk-Pose uses camera-pose similarity to shortlist historical chunks, then applies the same trained router to select sections within them. For this comparison, WorldKV uses its pose-based whole-chunk retrieval on our frozen backbone with matched budgets, without World Compression. We evaluated two additional noise seeds (three in total) on all 150 scenes with 180° rotation, demonstrating that MosaiChunk-Pose and WorldKV consistently achieve similar performance across seeds.

∥Fc∥\lVert\mathcal{F}_c\rVertSeedWorldKVMosaiChunk-Pose
1Original0.853 / 0.6100.843 / 0.624
1Additional 10.869 / 0.5940.861 / 0.600
1Additional 20.856 / 0.6050.851 / 0.598
2Original0.891 / 0.5780.893 / 0.577
2Additional 10.905 / 0.5370.902 / 0.545
2Additional 20.895 / 0.5540.894 / 0.566

Cells report median CLIP ↑ / LPIPS ↓.

Reviewer_QRP5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses long-term visual forgetting in autoregressive video generation: previously generated objects or scenes may lose their established appearance after leaving the sliding window. The proposed method, MosaiChunk, partitions historical key–value entries into equal-sized sections and uses learned descriptors to retrieve and compose sections from different spatial positions and historical chunks under a fixed active-KV budget. The video generator remains frozen, while a lightweight router is trained through self-distillation against the same backbone supplied with richer, oracle-selected historical context. Public benchmarks, ablations, and pose-guided comparisons provide additional evidence, although advantages over stronger retrieval baselines and across other camera trajectories are less consistent.

Critical strengths
  1. A meaningful visual-memory mechanism supported by targeted interventions. Section 3.1 and Appendix C.4 demonstrate that selected historical KV entries can restore content in a frozen generator, including transferring localized content between rollouts. These interventions provide meaningful evidence that historical KV can function as reusable, content-associated memory. This is a promising direction for extending video memory without updating the generator itself.
  2. Substantial improvements under a reasonably controlled active-memory budget. The comparisons match active visual KV usage: retrieval methods allocate part of the budget to distant history, while the sliding-window baseline retains additional recent chunks. Improvements across two backbones and additional public-benchmark evaluations support the practical value of the complete system in some revisit settings.
Critical weaknesses
  1. The independent benefit of fine-grained KV composition is not sufficiently isolated. In Table 13, once pose guidance is matched, MosaiChunk-Pose offers only marginal or inconsistent gains over whole-chunk WorldKV . Meanwhile, Table 4’s descriptor ablation also changes the pooled representation and value weighting. Without matched-component controls and uncertainty estimates, the evidence supports the full router pipeline rather than showing that section-level composition itself drives the improvement, which directly weakens the central novelty claim.
  2. The revisit evaluation does not cleanly isolate out-of-context memory recovery. In I2V, the departure frame remains in the permanent first-chunk attention sink, and translation revisit selection ignores camera position error (Appendix B.3). In T2V, results are conditioned on 100/466 pre-screened scenes and manual per-rollout frame selection without reporting per-method compliance, actual occlusion duration. This makes it difficult to attribute the reported improvements purely to long-horizon memory recovery.
Questions for authors
  1. Contribution attribution and stronger baselines — clarification and controlled evidence requested. Does section-level retrieval retain an advantage over whole-chunk retrieval when descriptor inputs, training objectives, value weighting, and available retrieval signals are matched? Please clarify which WorldKV components are included in the adaptation and provide paired uncertainty estimates for the comparisons in Table 13. An existing matched-component comparison, or a targeted additional control, would help establish whether fine-grained composition provides an independent benefit.
  2. I2V evaluation — protocol clarification requested. How does the evaluation distinguish recovery from distant history from recovery using the permanently retained first-chunk sink? Are results available for revisiting content introduced outside the sink? For translation trajectories, please report actual departure and return pose errors, reconstruction failures, and valid-revisit counts by method. Do the conclusions hold when evaluating the commanded endpoint or selecting revisits with both position and orientation constraints?
  3. T2V screening and annotation — clarification requested. Do all retained scenarios successfully disappear and reappear under every compared method and budget, and how are failures scored? Please report actual out-of-view durations and explain whether annotation was blinded to method identity. Does the shared departure-frame index correspond to a fully visible target in every rollout?
3 comments
AuthorsOpenReview ↗

Response to Reviewer QRP5 (1/3): Retrieval Ablations

Reply 1/3

We thank the reviewer for their thoughtful feedback. We appreciate their recognition of the KV interventions and the improvements under matched memory budgets. Below, we provide additional retrieval comparisons, and clarify and refine the evaluation protocols. We will incorporate these results and clarifications in a revised version.

Q1a. Does section composition help when retrieval components and budgets are matched?

Following the reviewer’s suggestion, we retrain whole-chunk routers for both I2V and T2V and find that they outperform MoC but fall short of MosaiChunk, as shown in the tables below. These results support the benefits of both learned descriptors and section composition, which MosaiChunk combines. Descriptor training encourages retrieval of relevant historical content over recent content, while section composition allows useful information from several historical chunks to share the same budget. Visual comparisons agree with the quantitative results. We will include these visualizations, including outputs from the retrained whole-chunk checkpoints, in a revised version.

We keep the descriptor architecture, teacher supervision, value weighting, and optimization settings unchanged. During training and inference, the router retrieves complete chunks ranked by their maximum section similarity to the query. Both I2V and T2V evaluations use a one-chunk far-memory budget. Due to time constraints, the I2V comparison focuses on 180° rotation.

I2V (180° rotation)

MethodLearned descriptorGranularityCLIP ↑LPIPS ↓
BaseNo—0.7680.655
MoCNoWhole0.8070.645
Whole-chunk (retrained)YesWhole0.8260.629
MosaiChunkYesSection0.8400.625

T2V

MethodLearned descriptorGranularityCLIP ↑LPIPS ↓
BaseNo—0.7550.659
MoCNoWhole0.8320.594
Whole-chunk (retrained)YesWhole0.8690.575
MosaiChunkYesSection0.8990.567

Additionally, MosaiChunk supports fractional-chunk budgets and does not require an integer number of chunks. MosaiChunk achieves comparable performance to retrained whole-chunk retrieval with half a chunk less memory (1.5 vs. 2 chunks).

∥Fc∥\lVert\mathcal{F}_c\rVertMoCWhole (retrained)MosaiChunk
0.5——0.815 / 0.643
1.00.807 / 0.6450.826 / 0.6290.840 / 0.625
1.5——0.850 / 0.618
2.00.807 / 0.6420.850 / 0.6140.863 / 0.609

Q1b. How is WorldKV adapted, and how consistent are the comparisons?

Existing pose-based retrieval mechanisms such as WorldKV’s require an input camera trajectory and are best suited to static scenes. In dynamic scenes, objects may move or change between visits, so retrieving memory from the same camera pose may return content that no longer matches the current scene. MosaiChunk instead retrieves memory by visual content without requiring camera trajectories, providing a general memory mechanism for KV-cached autoregressive video generation.

As stated in Appendix C.3, we do not claim a consistent advantage over WorldKV. This ablation examines whether pose can provide an additional retrieval signal for MosaiChunk. MosaiChunk-Pose uses camera-pose similarity to shortlist historical chunks, then applies the same trained router to select sections within them. For this comparison, WorldKV uses its pose-based whole-chunk retrieval on our frozen backbone with matched budgets, without World Compression. We evaluated two additional noise seeds (three in total) on all 150 scenes with 180° rotation, demonstrating that MosaiChunk-Pose and WorldKV consistently achieve similar performance across seeds.

∥Fc∥\lVert\mathcal{F}_c\rVertSeedWorldKVMosaiChunk-Pose
1Original0.853 / 0.6100.843 / 0.624
1Additional 10.869 / 0.5940.861 / 0.600
1Additional 20.856 / 0.6050.851 / 0.598
2Original0.891 / 0.5780.893 / 0.577
2Additional 10.905 / 0.5370.902 / 0.545
2Additional 20.895 / 0.5540.894 / 0.566

Cells report median CLIP ↑ / LPIPS ↓.

AuthorsOpenReview ↗

Response to Reviewer QRP5 (2/3): I2V Evaluation

Reply 2/3

Q2a. Do the gains persist for revisits outside the fixed attention sink?

Following the reviewer’s concern, we conduct an additional evaluation outside the fixed attention sink and find that MosaiChunk still achieves higher CLIP and lower LPIPS than Base and MoC at both budgets, for rotation with and without translation.

Following WorldTrace, we calculate CLIP and LPIPS over five departure–revisit frame pairs, all outside the sink chunk. The departure times are 1, 1.5, 2, 2.5, and 3 seconds. We match return frames using estimated camera poses and average the scores over the five pairs in each scene, then across all scenes. Each table cell reports average CLIP ↑ / LPIPS ↓ scores.

Trajectory∥Fc∥\lVert\mathcal{F}_c\rVertBaseMoCMosaiChunk
Rotation10.813 / 0.6340.829 / 0.6180.838 / 0.614
Rotation20.837 / 0.6220.842 / 0.6150.870 / 0.593
Rotation + translation10.831 / 0.6080.847 / 0.6010.857 / 0.594
Rotation + translation20.842 / 0.6050.857 / 0.6070.872 / 0.594

Q2b. How do camera-pose errors affect the translation evaluation?

Compared with trajectories in existing benchmarks of video models' camera controllability, our long-horizon trajectories combining 180° rotation and translation are much more challenging. Median departure–return errors are approximately 11–15° in rotation and 27–31% of the maximum reconstructed displacement in position. All reconstructions produced numerically valid poses, although scene inconsistency can affect their accuracy.

To mitigate the effect of imperfect revisits, we recompute CLIP and LPIPS on scenes with better estimated camera controllability. Following PoseDiffusion, we measure angular errors in relative rotation and translation direction between Pi3's reconstructed and commanded trajectories. For each video, we check frame pairs along its camera trajectory. More than half of the evaluated pairs must have both errors below 20°. We keep a scene only when Base, MoC, and MosaiChunk all meet this requirement. This leaves 44 of 100 scenes at budget 1 and 34 of 100 at budget 2. We also evaluate all scenes at the commanded endpoint, whose frame index does not use reconstructed poses. MosaiChunk retains higher median CLIP and lower median LPIPS in both evaluations.

∥Fc∥\lVert\mathcal{F}_c\rVertEvaluationBaseMoCMosaiChunk
120° Threshold0.788 / 0.6530.805 / 0.6200.854 / 0.618
1Commanded endpoint0.753 / 0.6630.783 / 0.6530.802 / 0.642
220° Threshold0.777 / 0.6200.800 / 0.6290.865 / 0.589
2Commanded endpoint0.776 / 0.6580.775 / 0.6620.813 / 0.642

Cells report median CLIP ↑ / LPIPS ↓.

To further reduce dependence on accurate camera pose estimates, we introduce an additional metric based on visual matches and their 3D consistency. We find LoFTR matches for each departure–revisit pair and obtain their 3D coordinates from Pi3. As in Map-free Visual Relocalization, we use RANSAC to estimate a single rotation and translation that aligns the matched points. Matches whose paired 3D points fall within a distance threshold after alignment are counted as 3D inliers.

∥Fc∥\lVert\mathcal{F}_c\rVertBaseMoCMosaiChunk
190.640 / 22.450135.540 / 40.910201.820 / 61.250
2121.630 / 34.100181.770 / 52.210297.110 / 108.220

Cells report mean LoFTR matches ↑ / 3D inliers ↑ over all 100 translation scenes.

MosaiChunk yields more visual matches and 3D inliers at both budgets. We also inspected correspondence visualizations to check that the matches align with recognizable scene structures. We will include the visualizations in a revised version.

AuthorsOpenReview ↗

Response to Reviewer QRP5 (3/3): T2V Evaluation

Reply 3/3

Q3. How are T2V scenes screened, revisit frames annotated, and failures handled?

Clarification of the T2V evaluation protocol: The RAVEN-adapted MiniMax-H3 LoRA is described by its authors as undertrained, and we observed scene-dependent instruction-following capabilities. We screened 466 base-model rollouts and retained 100 that followed the action sequence, using the same prompts and initial noise across methods. Since action timing varies, we originally selected frames manually with method identities visible. Targets were fully visible in the departure frames. When an object did not fully reappear, we selected the revisit frame where it was most visible, which could lower the consistency scores.

To reduce potential bias from manual frame selection, we replace it with automatic, method-blind VLM annotation and find that the results remain consistent with those reported in the paper. MosaiChunk still outperforms Base and achieves higher CLIP than MoC, with scores close to the original results. We calculate CLIP and LPIPS only on valid revisits; invalid revisits are counted but excluded from metric calculation. We will update all T2V evaluation results using VLM-selected frame pairs in the revised version.

We use a VLM (Qwen3.8-Flash-Next-FP8) with two annotated examples to annotate all 100 scenes.

  • Input: Video frames sampled every 0.25 seconds, with instructions to identify disappearance and return. Method names are hidden.
  • Output: Whether a valid revisit occurs and the departure/revisit frame indices, selected separately for each rollout.

Valid pairs contain comparable departure and revisit frames where the objects went out of view. We also manually inspected the VLM-selected departure and revisit frames and confirmed their validity.

MethodValid pairs / 100Out-of-view duration (s)VLM CLIP ↑ / LPIPS ↓Paper CLIP ↑ / LPIPS ↓
Base903.5000.768 / 0.6630.755 / 0.659
MoC923.6880.834 / 0.5700.832 / 0.594
MosaiChunk944.0210.871 / 0.5740.899 / 0.567
Reviewer_LVpkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MosaiChunk, a memory mechanism for autoregressive video generation that retrieves and composes fine-grained historical KV sections under a fixed active-cache budget. The video backbone remains frozen, while a lightweight router is trained through self-distillation using richer, oracle historical context. The authors also introduce RememBench and evaluate revisit consistency across text-to-video and image-to-video settings.

Critical strengths
  1. Preserving visual details when objects or scenes reappear is an important problem in long-horizon video generation. The paper motivates this problem clearly, and the illustrations make the proposed memory architecture and training procedure easy to follow.
  2. The observation that a frozen generator can reuse non-contiguous historical KV provides a natural foundation for the method.
  3. The evaluation includes two backbones, T2V and I2V tasks, different memory budgets, architectural ablations, and public benchmarks. The improvements over the swa baselines are encouraging.
Critical weaknesses
  1. The method changes the visible context and temporal positional encoding while keeping the backbone frozen. Quantitative oracle-memory and full-history comparisons, together with a positional-remapping ablation, would clarify whether the gains come from recovering information, filtering interference, or adjusting positions.
  2. Evaluation protocol needs clarification. T2V retains 100 of 466 candidate scenes and uses manually selected revisit frames, raising questions about generality and annotation consistency. I2V evaluates against the initial image while retaining the first chunk as an attention sink. Testing later-introduced content that has left all active context would more directly assess long-term retrieval.
  3. Although active KV usage is bounded, the CPU history bank grows with rollout length. Memory consumption and end-to-end latency, including clustering, retrieval, and CPU–GPU transfer, are needed to establish practical scalability.
Questions for authors
  1. Could you compare section-level and whole-chunk retrieval using similarly trained routers, the same weighting mechanism, and matched memory budgets? How should the small gains over pose-guided WorldKV in Table 13 be interpreted?
  2. Could you provide quantitative oracle-memory and full-history baselines, at least on manageable sequence lengths, and ablate temporal RoPE remapping?
  3. Could you evaluate revisits to content outside the fixed attention sink, and clarify the T2V annotation protocol and action-compliance rates across methods?
  4. Could you report CPU/GPU memory usage and end-to-end generation latency as history length increases, including retrieval and transfer overhead?
4 comments
AuthorsOpenReview ↗

Response to Reviewer LVpk (1/3): Retrieval Ablations

Reply 1/3

We thank the reviewer for their thoughtful feedback. We appreciate their recognition of the importance of long-term visual memory and our evaluation across two backbones. Below, we provide additional ablations and profiling results, and clarify and refine the evaluation protocols. We will incorporate these results and clarifications in a revised version.

Q1a. Does section-level retrieval improve over similarly trained whole-chunk retrieval?

Following the reviewer’s suggestion, we retrain whole-chunk routers for both I2V and T2V and find that they outperform MoC but fall short of MosaiChunk, as shown in the tables below. These results support the benefits of both learned descriptors and section composition, which MosaiChunk combines. Descriptor training encourages retrieval of relevant historical content over recent content, while section composition allows useful information from several historical chunks to share the same budget. Visual comparisons agree with the quantitative results. We will include these visualizations, including outputs from the retrained whole-chunk checkpoints, in a revised version.

We keep the descriptor architecture, teacher supervision, value weighting, and optimization settings unchanged. During training and inference, the router retrieves complete chunks ranked by their maximum section similarity to the query. Both I2V and T2V evaluations use a one-chunk far-memory budget. Due to time constraints, the I2V comparison focuses on 180° rotation.

I2V (180° rotation)

MethodLearned descriptorGranularityCLIP ↑LPIPS ↓
BaseNo—0.7680.655
MoCNoWhole0.8070.645
Whole-chunk (retrained)YesWhole0.8260.629
MosaiChunkYesSection0.8400.625

T2V

MethodLearned descriptorGranularityCLIP ↑LPIPS ↓
BaseNo—0.7550.659
MoCNoWhole0.8320.594
Whole-chunk (retrained)YesWhole0.8690.575
MosaiChunkYesSection0.8990.567

Additionally, MosaiChunk supports fractional-chunk budgets and does not require an integer number of chunks. MosaiChunk achieves comparable performance to retrained whole-chunk retrieval with half a chunk less memory (1.5 vs. 2 chunks).

∥Fc∥\lVert\mathcal{F}_c\rVertMoCWhole (retrained)MosaiChunk
0.5——0.815 / 0.643
1.00.807 / 0.6450.826 / 0.6290.840 / 0.625
1.5——0.850 / 0.618
2.00.807 / 0.6420.850 / 0.6140.863 / 0.609

Q1b. How should the small gains over pose-guided WorldKV be interpreted?

Existing pose-based retrieval mechanisms such as WorldKV’s require an input camera trajectory and are best suited to static scenes. In dynamic scenes, objects may move or change between visits, so retrieving memory from the same camera pose may return content that no longer matches the current scene. MosaiChunk instead retrieves memory by visual content without requiring camera trajectories, providing a general memory mechanism for KV-cached autoregressive video generation.

As stated in Appendix C.3, we do not claim a consistent advantage over WorldKV. This ablation examines whether pose can provide an additional retrieval signal for MosaiChunk. MosaiChunk-Pose uses camera-pose similarity to shortlist historical chunks, then applies the same trained router to select sections within them. For this comparison, WorldKV uses its pose-based whole-chunk retrieval on our frozen backbone with matched budgets, without World Compression. We evaluated two additional noise seeds (three in total) on all 150 scenes with 180° rotation, demonstrating that MosaiChunk-Pose and WorldKV consistently achieve similar performance across seeds.

∥Fc∥\lVert\mathcal{F}_c\rVertSeedWorldKVMosaiChunk-Pose
1Original0.853 / 0.6100.843 / 0.624
1Additional 10.869 / 0.5940.861 / 0.600
1Additional 20.856 / 0.6050.851 / 0.598
2Original0.891 / 0.5780.893 / 0.577
2Additional 10.905 / 0.5370.902 / 0.545
2Additional 20.895 / 0.5540.894 / 0.566

Cells report median CLIP ↑ / LPIPS ↓.

AuthorsOpenReview ↗

Response to Reviewer LVpk (2/3): Memory and Evaluation

Reply 2/3

Q2. What do the Oracle, Full history, and temporal RoPE remapping ablations show?

We add Oracle and Full history baselines and a temporal RoPE remapping ablation on all 150 I2V scenes with 180° rotation. Section and Oracle use a one-chunk far-memory budget. Oracle selects historical chunks closest to the commanded camera pose. Full history retains all previous chunks.

MethodRoPE remappingSink CLIP ↑Sink LPIPS ↓
Base—0.7680.655
SectionNo0.8400.609
Section (Ours)Yes0.8400.625
OracleNo0.8370.602
OracleYes0.8670.593
Full history—0.8820.603

Base performs worst because it has no far memory. Directly providing historical memory during denoising, through Oracle or Full history, substantially improves generation consistency. This suggests that recovering relevant information is the key factor. Section approaches Oracle at the same one-chunk budget and substantially outperforms Base.

We follow existing work such as WorldPlay and WorldTrace to remap retrieved keys to their assigned temporal positions. We further ablate this adjustment to evaluate its effect. The Oracle results show its benefit: CLIP increases from 0.837 to 0.867, and LPIPS decreases from 0.602 to 0.593.

Q3a. Do the gains persist for revisits outside the fixed attention sink?

Following the reviewer’s concern, we conduct an additional evaluation outside the fixed attention sink and find that MosaiChunk still achieves higher CLIP and lower LPIPS than Base and MoC at both budgets, for rotation with and without translation.

Following WorldTrace, we calculate CLIP and LPIPS over five departure–revisit frame pairs, all outside the sink chunk. The departure times are 1, 1.5, 2, 2.5, and 3 seconds. We match return frames using estimated camera poses and average the scores over the five pairs in each scene, then across all scenes. Each table cell reports average CLIP ↑ / LPIPS ↓ scores.

Trajectory∥Fc∥\lVert\mathcal{F}_c\rVertBaseMoCMosaiChunk
Rotation10.813 / 0.6340.829 / 0.6180.838 / 0.614
Rotation20.837 / 0.6220.842 / 0.6150.870 / 0.593
Rotation + translation10.831 / 0.6080.847 / 0.6010.857 / 0.594
Rotation + translation20.842 / 0.6050.857 / 0.6070.872 / 0.594

Q3b. How are T2V revisit frames selected and action compliance evaluated?

Clarification of the T2V evaluation protocol: The RAVEN-adapted MiniMax-H3 LoRA is described by its authors as undertrained, and we observed scene-dependent instruction-following capabilities. We screened 466 base-model rollouts and retained 100 that followed the action sequence, using the same prompts and initial noise across methods. Since action timing varies, we originally selected frames manually with method identities visible. Targets were fully visible in the departure frames. When an object did not fully reappear, we selected the revisit frame where it was most visible, which could lower the consistency scores.

To reduce potential bias from manual frame selection, we replace it with automatic, method-blind VLM annotation and find that the results remain consistent with those reported in the paper. MosaiChunk still outperforms Base and achieves higher CLIP than MoC, with scores close to the original results. We calculate CLIP and LPIPS only on valid revisits; invalid revisits are counted but excluded from metric calculation. We will update all T2V evaluation results using VLM-selected frame pairs in the revised version.

We use a VLM (Qwen3.8-Flash-Next-FP8) with two annotated examples to annotate all 100 scenes.

  • Input: Video frames sampled every 0.25 seconds, with instructions to identify disappearance and return. Method names are hidden.
  • Output: Whether a valid revisit occurs and the departure/revisit frame indices, selected separately for each rollout.

Valid pairs contain comparable departure and revisit frames where the objects went out of view. We also manually inspected the VLM-selected departure and revisit frames and confirmed their validity.

MethodValid pairs / 100Out-of-view duration (s)VLM CLIP ↑ / LPIPS ↓Paper CLIP ↑ / LPIPS ↓
Base903.5000.768 / 0.6630.755 / 0.659
MoC923.6880.834 / 0.5700.832 / 0.594
MosaiChunk944.0210.871 / 0.5740.899 / 0.567
AuthorsOpenReview ↗

Response to Reviewer LVpk (3/3): Profiling

Reply 3/3

Q4. How do CPU/GPU memory usage and generation latency scale with history length?

We profile 8 I2V and 6 T2V scenes on H200 GPUs, excluding warmup and initial model loading. I2V uses one GPU per video; T2V uses four-way sequence parallelism with eight-way parameter sharding across two concurrent videos. Chunk latency includes denoising, clustering, retrieval, and CPU–GPU transfers.

I2V

∥Fc∥\lVert\mathcal{F}_c\rVertHistory (chunks)CPU RAM (GiB/video)GPU (GiB/card)Router off (s/chunk)Router on (s/chunk)Overhead
1631.0100.85.126.2021.1%
11049.9100.85.176.5025.7%
11469.1100.85.156.3122.5%
2634.9110.45.276.4121.7%
21054.0110.45.287.2036.4%
21473.5110.55.367.2635.4%

Mean full-video latency (253 frames), router off → on: 109.78 → 142.12 s at budget 1; 124.90 → 176.27 s at budget 2.

T2V

∥Fc∥\lVert\mathcal{F}_c\rVertHistory (chunks)CPU RAM (GiB/video)GPU (GiB/card)Router off (s/chunk)Router on (s/chunk)Overhead
16272.944.22.844.5459.9%
112311.244.32.824.8873.2%
120367.044.42.815.5095.5%
26287.449.42.463.9560.5%
212329.249.52.474.5785.1%
220384.949.52.464.5986.4%

Mean full-video latency (379 frames), router off → on: 111.08 → 161.62 s at budget 1; 111.30 → 151.17 s at budget 2.

At a fixed far-memory budget, GPU memory remains nearly constant as history grows, while CPU memory increases. The relative per-chunk latency overhead is higher for T2V than for I2V. Our implementation is not yet fully optimized at the systems level; further optimization of memory transfers and parallel execution may reduce latency.

Thanks for the rebuttal. I maintain my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions

We thank all reviewers for their thoughtful feedback. As less than a day remains in the discussion period, we would greatly appreciate hearing from each reviewer whether our responses and revisions have addressed your concerns. We are happy to clarify anything further before the deadline.

We have uploaded a revised paper with all changes marked in blue. The main changes are:

  1. Retrained whole-chunk baseline (all reviewers): uses the same descriptor, teacher supervision, and value weighting as MosaiChunk (Table 4, Fig. 11).
  2. Evaluation protocol (LVpk, QRP5): method-blind VLM annotation for T2V (Table 1, App. B.3); I2V revisits outside the attention sink, sensitivity to camera control, and a LoFTR/3D-inlier metric (App. C.2, Tables 16–18).
  3. Oracle, full-history, and RoPE remapping ablations (LVpk): App. C.3, Table 19.
  4. Memory and latency profiling (LVpk): App. A.3, Tables 7–8.
  5. WorldKV comparison over three seeds (all reviewers): App. C.3, Table 21.
  6. Related work (otnX): discussion of SlotMemory and LayerRecall (Sec. 2).

Thank you again for your time and engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.