acceptodds
Under review as a conference paper at ICLR 2027

Learning a Spatial Memory Codec for Video World Model

Abstract

Generative video world models are emerging as a foundation for interactive agents, simulators, and creative tools that synthesize visual trajectories through spatial environments. Scaling such models to long trajectories requires more than local temporal coherence: when a place is revisited, the model must recover visual information from much earlier observations. Since keeping all observations in the model's context is not scalable, the model needs a long-term memory that preserves useful visual information while keeping storage compact and retrieval efficient. Existing approaches summarize history into a fixed-size state, retain a growing archive, or rely on explicit 3D reconstruction. These choices can lose important visual information, incur high storage and retrieval costs, or require hand-crafted update rules. We introduce a learned spatial memory codec that stores history in compact learned entries. These entries are retrieved by a reader through a hierarchical, pose-indexed mechanism that efficiently selects those most relevant to each generated view. Specifically, each video chunk is compressed into a small set of entries, each paired with a compact index representation that we call a landmark. Each landmark summarizes its corresponding entry and records where and when it was observed. To form the generation context, the reader first selects a fixed-size set of the most relevant landmarks and then uses only their corresponding entries. We show that our memory codec balances compact storage, efficient retrieval, and consistent generation across challenging revisits, including viewpoint changes and full U-turns. We further demonstrate retrieval from nearly two hours of observations, far beyond the training horizon.

Primary area:
generative models
Keywords:
Diffusion Models, Video Generation, Long Video Generation, World Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m5oFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors notice the challenge of maintaining long-term spatial consistency in generative video world models, requiring representations that are compact, informative, and efficiently retrievable. Therefore, the authors propose a learned spatial memory codec that compresses video chunks into memory entries, each associated with a lightweight landmark containing a learned content summary and camera pose and time information. Given a target viewpoint, Sparse Landmark Attention selects a small set of relevant entries and decodes them into features, and an adaptive allocation mechanism further adjusts the number of stored entries according to the requested reconstruction quality. Experiments show improved revisit consistency over existing baselines and effective retrieval from observation archives spanning nearly two hours, while keeping the expensive decoding stage bounded by a fixed read budget.

Critical strengths
  1. The paper addresses an essential problem, maintaining spatial consistency when the generative world model revisits previously observed locations. The proposed method efficiently retrieves memory entries associated with relevant past views and guides the target view to generate, which is well motivated. Especially, using compact, pose-aware landmards as a clue provides a pratical way to recover visual information while keeping the decoding cost budget low.

  2. The paper is clearly written and well organized. The figures effectively illustrate the proposed framework and training pipeline, while the tables clearly present the experimental results, making the method and its evaluation easy to follow.

  3. The experimental evaluation is comprehensive, covering memory compression, retrieval strategies, adaptive storage allocation, and end-to-end video generation

Critical weaknesses
  1. The benefit of learned retrieval over simple pose-based selection appears limited. As shown in Table 1(b), the proposed SLA-Select improves PSNR from 20.17 dB with nearest-pose selection to 20.28 dB, a gain of only 0.11 dB. Given the additional complexity of training the retrieval mechanism, further evidence would help establish when learned retrieval provides meaningful benefits and justify its use over the simpler baseline.

  2. The role of learned content representations in retrieval is insufficiently justified. Since the landmark-matching query is derived from the target camera geometry, it remains unclear what discriminative information the content summaries provide beyond geometric proximity and orientation. This concern is reinforced by Table 1(b), where learned retrieval improves PSNR over nearest-pose selection by only 0.11 dB.

  3. The paper does not systematically examine how the benefits of the proposed memory mechanism vary with model size or context length. Although the results support its effectiveness in the evaluated configuration, it remains unclear whether comparable gains could be achieved by allocating similar resources to a larger backbone or a longer context window.

Questions for authors

Robustness to erroneous history remains unclear. Although the paper evaluates retrieval under unrelated observations, this does not directly address spatially relevant but unreliable entries, such as corrupted observations, inaccurate pose annotations, or inconsistent generated content

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (m5oF)

We thank the reviewer for their thoughtful feedback. We are glad they appreciated the motivation behind our approach and the efficient design of our memory retrieval mechanism. We also appreciate their positive assessment of the paper's clarity and organization, as well as their recognition of our comprehensive experimental evaluation. Below, we address the concerns raised by the reviewer.

Learned retrieval against nearest pose (W1)

We thank the reviewer for raising this important point. We clarify that Table 1b of the submitted paper compares selection strategies at inference using a model trained with SLA-Select. Therefore, even when nearest-pose selection replaces SLA-Select at inference, the encoder still produces memory representations that were optimized jointly with learned retrieval. Thus, Table 1b does not fully capture the benefits of joint optimization of the memory representations and retrieval policy compared to using a fixed retrieval rule.

To further investigate this, we extended Table 1b to include two additional experimental settings. First, we evaluated nearest-pose selection both as an inference-only replacement (as in the original table) and by separately fine-tuning the codec with nearest-pose retrieval (bottom part of the table). Our learned selection consistently outperforms both variants, with a larger advantage over the fine-tuned nearest-pose model. Second, we evaluated multiple read budgets (K=4,8,12K=4, 8, 12). As reconstruction becomes more sensitive to the selection policy when fewer entries are retrieved, the advantage of learned selection increases as KK decreases. Compared with the fine-tuned nearest-pose model, SLA-Select improves PSNR by 0.64 dB at K=4K=4, 0.45 dB at K=8K=8, and 0.34 dB at K=12K=12. Moreover, SLA-Select nearly matches the dense-attention oracle across all tested read budgets. Following a request from another reviewer, we also expanded the evaluation set, and the entire revised Table 1b is now evaluated on a clean set of DL3DV scenes.

What does the learned summary of the landmark add above the time and pose? (W2)

The learned summary allows the encoder to influence retrieval based on the content of each memory entry. Without it, retrieval would depend only on pose and time, and the encoder could not communicate which entries are most useful for reconstructing a target view. For example, observations with similar poses may differ substantially in usefulness due to blur or occlusions. By deriving the landmark summary from the encoded entry and optimizing it jointly with retrieval, our method can learn to account for such differences. Our ablations in the submitted paper and additional experiments in the rebuttal demonstrate the advantages of learned retrieval over nearest-pose selection.

Comparison with larger backbones and longer context windows (W3)

Our goal is to support long-term visual memory under the efficiency constraints of real-time video generation. Increasing backbone size or context length incurs additional computational and memory costs, making these alternatives challenging under real-time constraints. Moreover, increasing backbone capacity may improve generation quality, but does not by itself address the challenge of retrieving relevant observations from histories that exceed the generator's context window. Our experiments demonstrate retrieval from nearly two hours of observations (Figure 5), which would be prohibitively expensive to retain directly in the context of a video generator. Our learned memory instead maintains a growing archive while keeping the expensive decoding stage and generator conditioning independent of archive length. We also compare against systems employing larger video-generation backbones, including AlayaWorld, which uses an approximately 13B-parameter LTX-based video model compared with our 5B-parameter Wan backbone. Despite this difference, our method achieves better generation quality (Table 2).

Robustness to erroneous history (Q1)

We conducted additional experiments to evaluate robustness to unreliable memory, both in reconstruction and video generation:

  • Corrupted observations. We applied blur, noise, JPEG compression, and occlusions to stored memory chunks. Generation PSNR decreases by at most 0.6 dB, while multi-view consistency, measured by MEt3R, remains largely unchanged.
  • Inaccurate poses. Perturbing the stored camera poses has a larger impact, reducing generation PSNR by 1.4–1.9 dB. We added this sensitivity to our limitations discussion.
  • Self-generated history. We evaluated autoregressive generation in which the model stores its own generated frames and subsequently revisits the same trajectory. Compared with using only the original observed history, incorporating self-generated observations improves return-view PSNR by 1.4–1.5 dB. These improvements extend beyond the generator's short temporal context, demonstrating that the memory mechanism can successfully reuse imperfect self-generated observations to improve future views.

As discussed in our limitations, preventing drift in autoregressive video generation is a broader research challenge beyond the scope of our memory mechanism. Nevertheless, these experiments demonstrate that our memory can effectively reuse imperfect observations, while identifying pose inaccuracies as an important remaining limitation.

Dear Reviewer,

Thank you again for your time and effort.

We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.

We would be happy to address any remaining concerns or provide further clarification. If our responses have addressed your remaining concerns, we would greatly appreciate it if you could consider strengthening your support for the paper.

Thank you!

Reviewer_Ai7AOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a learned external memory for camera-conditioned video generation: each 8-frame chunk is compressed into a few variable-length entries, each indexed by a compact landmark holding pose, time and a content summary. At query time, a reader scores all landmarks, decodes only the top-K entries into target-view features, and feeds them to a Wan2.2-5B model. The codec is evaluated on its own for held-out-view reconstruction and on KITTI-360 archives of up to ~2 hours, and the full model is compared with six released world models on 32 revisit clips.

Critical strengths
  • It addresses a central bottleneck: bounded-cost, long-horizon revisit memory for video world models. The design is clean and modular. Write-once compact entries, cheap landmarks for addressing, and a fixed read budget decouple storage growth from decode cost.
  • Scaling evidence is strong: flat PSNR as irrelevant history grows to ~2 h, dense reads hitting OOM, and concrete latency and storage numbers.
  • Revisits are tested on genuine loop trajectories, not only out-and-back palindromes.
  • Adaptive retention via nested dropout with a quality-conditioned allocation head is a sensible, novel use of variable-rate coding for memory.
Critical weaknesses
  • Some key comparisons are missing. No experiment uses the same Wan backbone and training with simple pose-based retrieval of raw frames or VAE latents (WorldMem / Context-as-Memory style). Without it, the gain of the learned codec over the non-learned retrieval it replaces is not isolated. The choice of a VGGT-Ω writer over the generator's own VAE latents is also neither motivated nor ablated. Prior use of VGGT features as implicit 3D scene context for a video DiT (e.g., CineScene, CVPR 2026) is not cited.
  • Learned retrieval adds little. Learned selection beats nearest-pose by only 0.11 dB (Table 1b). Is there any setting where pose-only selection should fail and learned selection should succeed?
  • The storage claim is only strong against a strawman. 12.6 GiB/h is compared to a full KV cache (~TB/h). The entries are larger than storing Wan VAE latents or compressed video. The real advantage is decode cost, not bytes.
Questions for authors
  1. The project page states that "SpatialVID, DL3DV and parts of Sekai have recorded training exposure", which covers up to 20 of the 32 Table-2 clips, but the paper does not mention this. Did any Table-2 clips, or their source videos or scenes, appear in training?
  2. How does the codec differ, conceptually and empirically, from ReWorld's pose-indexed landmark bank, RELIC's compressed camera-aware KV cache, and WorldTrace's summary slots?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (Ai7A)

We thank the reviewer for their thoughtful feedback. We are glad they appreciated the clean and modular design of our memory architecture, as well as its ability to support long-term memory with bounded decoding cost. We also appreciate their recognition of our evaluation on genuine loop trajectories and our novel approach to adaptive retention. Below, we address the concerns raised by the reviewer.

Learned retrieval against nearest pose (W1, W2)

We thank the reviewer for raising this important point. We clarify that Table 1b of the submitted paper compares selection strategies at inference using a model trained with SLA-Select. Therefore, even when nearest-pose selection replaces SLA-Select at inference, the encoder still produces memory representations that were optimized jointly with learned retrieval. Thus, Table 1b does not fully capture the benefits of joint optimization of the memory representations and retrieval policy compared to using a fixed retrieval rule.

To further investigate this, we extended Table 1b to include two additional experimental settings. First, we evaluated nearest-pose selection both as an inference-only replacement (as in the original table) and by separately fine-tuning the codec with nearest-pose retrieval (bottom part of the table). Our learned selection consistently outperforms both variants, with a larger advantage over the fine-tuned nearest-pose model. Second, we evaluated multiple read budgets (K=4,8,12K=4, 8, 12). As reconstruction becomes more sensitive to the selection policy when fewer entries are retrieved, the advantage of learned selection increases as KK decreases. Compared with the fine-tuned nearest-pose model, SLA-Select improves PSNR by 0.64 dB at K=4K=4, 0.45 dB at K=8K=8, and 0.34 dB at K=12K=12. Moreover, SLA-Select nearly matches the dense-attention oracle across all tested read budgets. Following a request from another reviewer, we also expanded the evaluation set, and the entire revised Table 1b is now evaluated on a clean set of DL3DV scenes.

A setting where pose-only selection should fail and learned selection should succeed (W2)

As discussed above, the advantage of learned retrieval increases as the read budget (KK) decreases. Unlike nearest-pose selection, our approach allows the encoder to influence which entries are retrieved through learned relevance scores, jointly optimizing the memory representations and selection policy for reconstruction. For example, consider two observations captured from similar camera poses, where one is blurry or contains a nearby object occluding much of the scene. A pose-only retrieval rule cannot distinguish between them based on their visual content. In contrast, our learned landmark summaries are derived from the encoded memory entries, allowing the encoder to communicate their usefulness for reconstructing a target view. This enables retrieval to account for factors beyond geometric proximity, such as blur and occlusions.

Comparison with raw-frame and VAE-latent memories (W1)

Our goal is to support real-time video generation while balancing generation quality, retrieval cost, and long-term storage. Raw-frame and VAE-latent memories face important tradeoffs in how historical observations are incorporated into the generator.

One approach, used by methods such as Context-as-Memory (CaM), incorporates retrieved observations into the generator's self-attention. Without caching, these observations must be repeatedly processed during generation, increasing computational cost and making real-time generation challenging. Alternatively, caching their KV representations reduces computation but requires substantially more storage per retained observation, making storage of long histories, such as our two-hour example, impractical. To quantify these tradeoffs, we added Table B.1, which includes both the original CaM results and a simulated CaM-style baseline using the same Wan generator and inference setup as ours. This controlled efficiency comparison demonstrates substantially faster generation and lower storage growth with our method, while maintaining comparable peak GPU memory usage.

Another approach is to incorporate retrieved frames or VAE latents through cross-attention. Prior Wan-based methods, including Matrix-Game 3.0 and DreamX-World, explored this alternative but reported lower generation quality than conditioning through self-attention. Our approach instead learns compact memory representations that can be efficiently retrieved and decoded into target-view features, avoiding the need to process historical observations directly within the generator.

Choice of VGGT-Ω\Omega as the writer backbone (W1)

During the development of our memory codec, we also experimented with using Wan VAE latents as the encoder representation. This variant achieved 15.6–16.1 dB validation PSNR, compared with 18.4 dB using VGGT-Ω\Omega features under the same evaluation setting. These results motivated our choice of VGGT-Ω\Omega as the writer backbone, as it provided a more effective representation for future-view reconstruction. We clarified this in the revision.

Missing reference (W1)

We thank the reviewer for the suggestion. We have added CineScene to the related work section.

Storage–retrieval–quality tradeoff (W3)

Our goal is to support real-time video generation, which requires balancing storage efficiency, retrieval cost, and generation quality. To better characterize this tradeoff, we added Table B.1, which compares generation speed, peak GPU memory usage, and storage growth across methods. As shown in this table, our method offers advantages in generation speed and storage efficiency over the compared systems, alongside the generation-quality improvements reported in Table 2, demonstrating a favorable tradeoff among these objectives.

Clarification of training and evaluation data overlap (Q1)

None of the evaluation clips used in our paper, nor their source videos or scenes, were used for training. The examples shown on the project page are from our validation set, which was used to guide high-level methodological decisions during development, consistent with standard validation practices, but not for training. To further strengthen evaluation independence, we added RealEstate10K (RE10K), which was not used during training or validation.

How does the codec differ, conceptually and empirically, from ReWorld's pose-indexed landmark bank, RELIC's compressed camera-aware KV cache, and WorldTrace's summary slots? (Q2)

ReWorld stores the generator's KV representations and retrieves historical chunks using a fixed pose-based rule. In contrast, our codec learns compact memory entries and jointly optimizes their representations and retrieval scores for reconstruction. Our ablations in the submitted paper and additional experiments in the rebuttal demonstrate the advantages of learned retrieval over nearest-pose selection, both in reconstruction and end-to-end video generation. We also directly compare against ReWorld, which uses the same Wan2.2-5B backbone, and achieve substantially better generation quality (Table 2).

RELIC compresses historical representations within the generator's KV cache, where attention performs both retrieval and aggregation over the compressed history. Consequently, attention cost grows with the retained history, potentially limiting scalability to very long archives. Our method instead learns a separate memory representation and explicitly separates lightweight landmark-based selection from the more expensive decoding of selected entries, keeping the latter independent of archive length. Since RELIC has not released a public implementation or checkpoint, we cannot directly compare generation performance.

WorldTrace compresses historical information into a bounded memory of summary slots and selected observations, merging or evicting entries to maintain a fixed capacity. In contrast, our method maintains a growing archive of learned entries and uses lightweight landmarks to retrieve relevant information. This enables efficient retrieval from nearly two hours of observations without merging or discarding earlier archived entries (Figure 5). WorldTrace does not provide a publicly available implementation, preventing a direct experimental comparison.

Dear Reviewer,

Thank you again for your time and effort.

We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.

We would be happy to address any remaining concerns or provide further clarification. If our responses have addressed your remaining concerns, we would greatly appreciate it if you could consider strengthening your support for the paper.

Thank you!

Reviewer_oMgNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a learned spatial memory codec for video world models. A geometry-aware writer compresses observed video chunks into ordered memory entries, while an allocation head predicts how many entries to retain for a requested reconstruction quality. Each entry has a compact landmark containing learned summaries and camera metadata. Sparse Landmark Attention selects relevant entries from the archive and decodes them into target-view features, which condition a video diffusion model through predicted VAE latents. The codec is pretrained through future-view reconstruction and subsequently trained jointly with the video generator. Experiments evaluate reconstruction quality, storage, retrieval from large observation archives, and memory-conditioned video generation.

Critical strengths
  1. The architecture addresses storage and retrieval jointly. Separating compact landmark indices from larger memory entries provides a practical way to search a growing archive while limiting expensive decoding. The controlled codec experiments show that compressed entries and sparse reading can preserve reconstruction quality while substantially reducing the number of tokens accessed. This is useful evidence beyond qualitative examples of successful revisits.

  2. The paper provides meaningful characterization of large-archive retrieval. The KITTI-360 experiments test fixed queries while adding unrelated observations, including archives approaching two hours across independent drives. Retrieval remains effective under these distractors. The paper also reports storage and resident-query timing, and explicitly distinguishes these measurements from end-to-end video generation throughput.

Critical weaknesses
  1. The end-to-end experiments do not sufficiently isolate the contribution of the proposed memory mechanisms. The main video-generation comparison uses 32 fixed clips, and Appendix B.5 acknowledges that training data, compute, native resolution, and sampling schedules differ across methods. Although the reported improvements are promising, a comparison using the same generator and training setup is needed to attribute them to the proposed codec and retrieval design. The codec ablations make this particularly relevant: nearest-pose selection achieves 20.17 dB versus 20.28 dB for learned SLA selection, while ordinary attention over learned-selected entries achieves 20.27 dB. These results support compact sparse memory, but leave the incremental value of the specific learned retrieval and decoding mechanisms less clearly established for video generation.

  2. Adaptive storage, a central contribution, lacks independent validation of its quality–storage tradeoff. Appendix F evaluates the allocation policy on 32 development clips also used for checkpoint monitoring and selection. Meanwhile, the principal codec comparison and archive-growth experiments retain all candidate entries, so those results do not establish the benefit of adaptive retention. It remains unclear whether the policy generalizes to unseen clips and improves the tradeoff over fixed retention or a simple redundancy-based policy at matched storage. In particular, the quality of generation after adaptive retention has not been adequately separated from the performance of the underlying full-retention codec.

Questions for authors
  1. Under the same Wan backbone, camera conditioning, data, and training budget, how does the proposed system compare with no external memory and with compressed memory using nearest-pose retrieval and ordinary decoding? Can you report both generation quality and resource usage to isolate the benefit of the learned retrieval mechanism?

  2. Can you evaluate adaptive retention on a test set unused for checkpoint selection, comparing against fixed retention and a simple redundancy-based policy at matched average storage? Please report achieved quality, storage, and requested-quality attainment relative to the quality attainable with full retention, including its effect on the integrated video generator.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (oMgN)

We thank the reviewer for their helpful feedback. We are glad they appreciated our joint approach to memory storage and retrieval, as well as the controlled experiments demonstrating its effectiveness. We also appreciate their recognition of our extensive evaluation of retrieval scalability over very long histories. Below, we address the concerns raised by the reviewer.

Fairness of video generation comparisons in Table 2 (W1)

As is standard practice for system-level evaluations, we compare against publicly released checkpoints of strong, recent methods using a shared evaluation protocol. Notably, some of these methods employ larger and potentially more powerful backbone models than ours. Training all competing methods under identical conditions is not practically feasible, particularly because they rely on different video-generation architectures (e.g., Wan, LTX, SANA, and Cosmos), and their approaches are not necessarily transferable across backbones. Table 2 evaluates the performance of complete systems, while our extensive controlled ablation studies isolate the contributions of the proposed components. These include the experiments already presented in the paper, as well as additional ablations provided in this rebuttal. Together, these evaluations demonstrate both the effectiveness of our complete system relative to existing methods and the importance of its individual components.

Expanded Evaluation in Table 2 (W1)

To strengthen the evaluation, we substantially increased the number of evaluated clips across all benchmarks and added RealEstate10K (RE10K), which was not used during training or validation. The updated results are presented in Table 2 of the revised paper.

Learned retrieval against nearest pose (W1)

We thank the reviewer for raising this important point. We clarify that Table 1b of the submitted paper compares selection strategies at inference using a model trained with SLA-Select. Therefore, even when nearest-pose selection replaces SLA-Select at inference, the encoder still produces memory representations that were optimized jointly with learned retrieval. Thus, Table 1b does not fully capture the benefits of joint optimization of the memory representations and retrieval policy compared to using a fixed retrieval rule.

To further investigate this, we extended Table 1b to include two additional experimental settings. First, we evaluated nearest-pose selection both as an inference-only replacement (as in the original table) and by separately fine-tuning the codec with nearest-pose retrieval (bottom part of the table). Our learned selection consistently outperforms both variants, with a larger advantage over the fine-tuned nearest-pose model. Second, we evaluated multiple read budgets (K=4,8,12K=4, 8, 12). As reconstruction becomes more sensitive to the selection policy when fewer entries are retrieved, the advantage of learned selection increases as KK decreases. Compared with the fine-tuned nearest-pose model, SLA-Select improves PSNR by 0.64 dB at K=4K=4, 0.45 dB at K=8K=8, and 0.34 dB at K=12K=12. Moreover, SLA-Select nearly matches the dense-attention oracle across all tested read budgets. Following the expansion of our evaluation discussed above, the entire revised Table 1b is now evaluated on a clean set of DL3DV scenes.

End-to-end ablation of external memory and learned retrieval (Q1)

To strengthen the evaluation of the learned retrieval, we conducted additional experiments using our autoregressive Wan generator on DL3DV scenes. We evaluated three configurations: (1) disabling external memory at inference, (2) replacing our learned retrieval with nearest-pose retrieval, and (3) our full method with learned SLA retrieval. All configurations use the same trained generator, with only the memory mechanism changed at inference. Results are shown in the table below. As can be seen, external memory substantially improves generation quality. Learned retrieval improves PSNR, LPIPS and rotation error over nearest-pose retrieval. Importantly, these improvements come at essentially identical inference speed and GPU memory usage.

AR variantPSNR↑LPIPS↓ΔRΩ\Delta R_\Omega (°)↓MEt3R↓FPS↑Peak GPU (GiB)↓
No memory11.600.5938.870.20641.722.5
Nearest-pose memory, K=4K=416.330.3281.380.21128.723.3
Ours (SLA-Select), K=4K=416.560.3191.290.21128.923.3

Evaluation of Adaptive Retention (W2, Q2)

An important aspect of our approach is that adaptive retention relies not only on the allocation policy, but also on the structure of the learned memory representations. Through nested dropout during training, we encourage earlier entries to preserve the most useful information, while later entries provide additional details. Without this ordering, selecting which entries to retain would require an additional selection mechanism or separate training for different compression rates. Our approach instead allows the number of retained entries to be adjusted through simple prefix truncation at inference, without retraining the codec. Importantly, this flexibility supports different retention policies, including fixed retention, which also benefits from the learned ordering.

To further evaluate adaptive retention, we replaced the evaluation set in Appendix F with a new evaluation on 48 held-out clips from RealEstate10K and Oxford Spires, neither of which was used for training or checkpoint selection. We compare adaptive retention against full retention and fixed retention at matched average storage, evaluating its effect on both memory reconstruction and video generation. Our results demonstrate substantial storage savings with minimal quality degradation. At 2.5× compression, reconstruction PSNR decreases by only 0.11 dB relative to full retention. Using the same retained entries, the bidirectional and autoregressive generators lose only 0.03 dB and 0.17 dB, respectively. Notably, the fixed-retention baseline also operates on our nested representations and therefore benefits from the learned ordering of entries. At matched storage, adaptive allocation further improves the quality–storage tradeoff on Oxford Spires while achieving comparable results on RealEstate10K (Figure F.1A). The per-chunk allocations in Figure F.1B illustrate how the policy distributes storage non-uniformly across observations. Together, these results demonstrate the benefits of learning a flexible memory representation that supports different retention policies, while showing that adaptive allocation can provide further improvements when memory requirements vary across observations.

Dear Reviewer,

Thank you again for your time and effort.

We hope that our rebuttal has addressed your concerns. With the discussion period nearing its end, we would appreciate hearing your thoughts.

We would be happy to address any remaining concerns or provide further clarification. If our responses have resolved your concerns, we would greatly appreciate it if you could reconsider your score.

Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful feedback and constructive suggestions. We appreciate their recognition of the importance of long-term memory for video world models, the design of our proposed memory architecture, and our experimental evaluation.

We would like to highlight three main points:

  1. The benefits of learned retrieval. All three reviewers raised concerns about the advantage of our learned retrieval mechanism over simpler nearest-pose selection. We clarified the original evaluation setting and why it did not fully capture the benefits of jointly learning the memory representations and retrieval mechanism. We also added new controlled experiments that better isolate these benefits and demonstrate when learned retrieval is particularly useful.
  2. Adaptive retention. We clarified that the contribution of our adaptive retention approach lies in both the learned memory representation, which enables flexible control over storage at inference, and the learned retention policy. We revised the evaluation, demonstrating that even simple retention policies benefit from our learned representation, while our adaptive policy can further improve the quality–storage tradeoff.
  3. Efficiency and real-time generation. We clarified that our goal is to support real-time video generation, which motivates our design choices and requires balancing generation quality, computational efficiency, and long-term storage. To better evaluate these tradeoffs, we added comparisons of generation speed, GPU memory usage, and storage growth against existing approaches.

Beyond these main points, we expanded the evaluation and conducted additional experiments addressing other concerns raised by the reviewers. We also clarified several methodological details and updated the related work and limitations discussions.

We hope that these additional experiments and clarifications address the reviewers' main concerns and strengthen the contributions of our work. We thank the reviewers and Area Chair again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.