- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors notice the challenge of maintaining long-term spatial consistency in generative video world models, requiring representations that are compact, informative, and efficiently retrievable. Therefore, the authors propose a learned spatial memory codec that compresses video chunks into memory entries, each associated with a lightweight landmark containing a learned content summary and camera pose and time information. Given a target viewpoint, Sparse Landmark Attention selects a small set of relevant entries and decodes them into features, and an adaptive allocation mechanism further adjusts the number of stored entries according to the requested reconstruction quality. Experiments show improved revisit consistency over existing baselines and effective retrieval from observation archives spanning nearly two hours, while keeping the expensive decoding stage bounded by a fixed read budget.
Critical strengths
-
The paper addresses an essential problem, maintaining spatial consistency when the generative world model revisits previously observed locations. The proposed method efficiently retrieves memory entries associated with relevant past views and guides the target view to generate, which is well motivated. Especially, using compact, pose-aware landmards as a clue provides a pratical way to recover visual information while keeping the decoding cost budget low.
-
The paper is clearly written and well organized. The figures effectively illustrate the proposed framework and training pipeline, while the tables clearly present the experimental results, making the method and its evaluation easy to follow.
-
The experimental evaluation is comprehensive, covering memory compression, retrieval strategies, adaptive storage allocation, and end-to-end video generation
Critical weaknesses
-
The benefit of learned retrieval over simple pose-based selection appears limited. As shown in Table 1(b), the proposed SLA-Select improves PSNR from 20.17 dB with nearest-pose selection to 20.28 dB, a gain of only 0.11 dB. Given the additional complexity of training the retrieval mechanism, further evidence would help establish when learned retrieval provides meaningful benefits and justify its use over the simpler baseline.
-
The role of learned content representations in retrieval is insufficiently justified. Since the landmark-matching query is derived from the target camera geometry, it remains unclear what discriminative information the content summaries provide beyond geometric proximity and orientation. This concern is reinforced by Table 1(b), where learned retrieval improves PSNR over nearest-pose selection by only 0.11 dB.
-
The paper does not systematically examine how the benefits of the proposed memory mechanism vary with model size or context length. Although the results support its effectiveness in the evaluated configuration, it remains unclear whether comparable gains could be achieved by allocating similar resources to a larger backbone or a longer context window.
Questions for authors
Robustness to erroneous history remains unclear. Although the paper evaluates retrieval under unrelated observations, this does not directly address spatially relevant but unreliable entries, such as corrupted observations, inaccurate pose annotations, or inconsistent generated content
3 comments
Robustness to erroneous history (Q1)
We conducted additional experiments to evaluate robustness to unreliable memory, both in reconstruction and video generation:
- Corrupted observations. We applied blur, noise, JPEG compression, and occlusions to stored memory chunks. Generation PSNR decreases by at most 0.6 dB, while multi-view consistency, measured by MEt3R, remains largely unchanged.
- Inaccurate poses. Perturbing the stored camera poses has a larger impact, reducing generation PSNR by 1.4–1.9 dB. We added this sensitivity to our limitations discussion.
- Self-generated history. We evaluated autoregressive generation in which the model stores its own generated frames and subsequently revisits the same trajectory. Compared with using only the original observed history, incorporating self-generated observations improves return-view PSNR by 1.4–1.5 dB. These improvements extend beyond the generator's short temporal context, demonstrating that the memory mechanism can successfully reuse imperfect self-generated observations to improve future views.
As discussed in our limitations, preventing drift in autoregressive video generation is a broader research challenge beyond the scope of our memory mechanism. Nevertheless, these experiments demonstrate that our memory can effectively reuse imperfect observations, while identifying pose inaccuracies as an important remaining limitation.
Dear Reviewer,
Thank you again for your time and effort.
We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.
We would be happy to address any remaining concerns or provide further clarification. If our responses have addressed your remaining concerns, we would greatly appreciate it if you could consider strengthening your support for the paper.
Thank you!
Response to Reviewer 3 (m5oF)
We thank the reviewer for their thoughtful feedback. We are glad they appreciated the motivation behind our approach and the efficient design of our memory retrieval mechanism. We also appreciate their positive assessment of the paper's clarity and organization, as well as their recognition of our comprehensive experimental evaluation. Below, we address the concerns raised by the reviewer.
Learned retrieval against nearest pose (W1)
We thank the reviewer for raising this important point. We clarify that Table 1b of the submitted paper compares selection strategies at inference using a model trained with SLA-Select. Therefore, even when nearest-pose selection replaces SLA-Select at inference, the encoder still produces memory representations that were optimized jointly with learned retrieval. Thus, Table 1b does not fully capture the benefits of joint optimization of the memory representations and retrieval policy compared to using a fixed retrieval rule.
To further investigate this, we extended Table 1b to include two additional experimental settings. First, we evaluated nearest-pose selection both as an inference-only replacement (as in the original table) and by separately fine-tuning the codec with nearest-pose retrieval (bottom part of the table). Our learned selection consistently outperforms both variants, with a larger advantage over the fine-tuned nearest-pose model. Second, we evaluated multiple read budgets (K=4,8,12). As reconstruction becomes more sensitive to the selection policy when fewer entries are retrieved, the advantage of learned selection increases as K decreases. Compared with the fine-tuned nearest-pose model, SLA-Select improves PSNR by 0.64 dB at K=4, 0.45 dB at K=8, and 0.34 dB at K=12. Moreover, SLA-Select nearly matches the dense-attention oracle across all tested read budgets. Following a request from another reviewer, we also expanded the evaluation set, and the entire revised Table 1b is now evaluated on a clean set of DL3DV scenes.
What does the learned summary of the landmark add above the time and pose? (W2)
The learned summary allows the encoder to influence retrieval based on the content of each memory entry. Without it, retrieval would depend only on pose and time, and the encoder could not communicate which entries are most useful for reconstructing a target view. For example, observations with similar poses may differ substantially in usefulness due to blur or occlusions. By deriving the landmark summary from the encoded entry and optimizing it jointly with retrieval, our method can learn to account for such differences. Our ablations in the submitted paper and additional experiments in the rebuttal demonstrate the advantages of learned retrieval over nearest-pose selection.
Comparison with larger backbones and longer context windows (W3)
Our goal is to support long-term visual memory under the efficiency constraints of real-time video generation. Increasing backbone size or context length incurs additional computational and memory costs, making these alternatives challenging under real-time constraints. Moreover, increasing backbone capacity may improve generation quality, but does not by itself address the challenge of retrieving relevant observations from histories that exceed the generator's context window. Our experiments demonstrate retrieval from nearly two hours of observations (Figure 5), which would be prohibitively expensive to retain directly in the context of a video generator. Our learned memory instead maintains a growing archive while keeping the expensive decoding stage and generator conditioning independent of archive length. We also compare against systems employing larger video-generation backbones, including AlayaWorld, which uses an approximately 13B-parameter LTX-based video model compared with our 5B-parameter Wan backbone. Despite this difference, our method achieves better generation quality (Table 2).