- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Paper proposes a method called SMART for long-horizon video generation, which first compresses each frame into multiple region tokens and writes them into a fixed number of slots based on camera parameters. When writing the memory into slots, only the selected slots are updated, while the other slots remain unchanged. When generating new frames, the system reads the memory using the target camera parameters and produces a coarse latent draft for the target view. The generated results are then written back into memory. Experiments on Memory Maze and Minecraft show that the method maintains good reconstruction quality and scene consistency over long videos.
Critical strengths
-
Slot Memory system greatly reduces the space needed to store memory while still providing good reconstruction quality.
-
Paper is well written and easy to understand. Besides, the experiments and ablations also validate the effectiveness of the proposed modules and the efficiency of the overall system.
Critical weaknesses
-
Concepts proposed in this paper are similar to those in existing work in the community, the core idea is to maintain a memory and use camera parameters to retrieve the required historical information. Besides, the paper does not directly compare proposed memory retrieval module with methods such as FOV IoU, the current experiments change several components at the same time, it is unclear how much of the improvement comes from the routing design itself.
-
All the experiments and visualizations in the paper use Memory Maze and Minecraft scenes ,which are relatively simple, and I am concerned that the method may perform differently in real-world scenes. Can the proposed slot memory recover fine details, such as text and patterns, in real-world scenes or more complex environments? I am also concerned about whether these low-level details are preserved in the slots and can be accurately retrieved.
Questions for authors
-
See Weaknesses 2.
-
Because the slot memory has limited capacity, will earlier memories fade after the model has seen many different scenes?
5 comments
Q3. Can the memory preserve fine details (text, patterns) in real-world scenes?
In line with prior art (Pasukonis et al., 2022; Po et al., 2025; Chandratreya et al., 2026; Xiao et al., 2026), we chose Memory Maze and Minecraft for our experiments, because they provide exact ground-truth poses, enforce long revisits, and come with unambiguous ground truth. They contain some details, but not as many as real-world clips (which are short and include few returns, offering little signal). Minecraft is open-terrain environment of size 256×256, with significant texture. In contrast to the baselines, SMART is able to consistently reconstruct details such as the sun position (Fig. 11). Regarding fine details, SMART is a coarse-to-fine system by design: the memory provides a coarse draft, which the diffusion transformer then refines based on its prior. The fidelity of the draft scales with capacity (Table 5).
We fully agree that real-world scenes are an important next step. For this first study, we had to stick with established benchmarks and concentrate on whether pose-addressed memory makes long-context video generation feasible. Scaling requires substantial compute resources to accommodate for both the memory capacity and the diffusion backbone. We simply did not have the computing resources to do this. We note that nothing in the addressing mechanism is specific to synthetic environments, and hope that future work building on ours will scale pose-addressed memory to real-world scenes.
Q4. Do earlier memories fade after the model has seen many scenes?
Within a scene, forgetting is tied to overwrites rather than to elapsed time: unselected slots receive neither decay nor writes, so a stored region fades when a new observation is routed to the same slot. Table 3 shows this: SMART loses only 0.8 dB from 401 to 3,201 frames, whereas a dense state of the same size loses 11.4 dB.
We thank the reviewer again and hope the new comparisons address the concerns. We will clarify our comparison with a camera-based retrieval method (GeoCache) and the position of SMART as an implicit, pose-addressed memory relative to explicit retrieval-based memories. We will also add the write-key ablation.
Thank you for your response and the additional experiments.
I have also considered the other reviewers' concerns and your responses, and I think some of the concerns have been addressed. However, the ability to faithfully preserve and recall fine details, especially in real-world scenes, remains unclear. It would be grate to include experiments on more complex scenes in the revised version.
For these reasons, I will maintain my original score of 2.
We thank the reviewer for the feedback and for noting the memory efficiency and clarity of the paper. We address the concerns below.
Q1. The core idea, maintaining a memory and using camera parameters to retrieve historical information, resembles existing work. What is conceptually new in SMART?
Of course we do not claim that using camera parameters to retrieve past content is new (Section 2). Our contribution is a learned, implicit memory for video, in which the camera determines how the memory is organized. Besides the conceptual novelty, this also brings a clear practical advance over explicit, retrieval-based memories.
In language modeling, implicit memory is well established. State-space models (Gu et al., 2021; Gu & Dao, 2023), gated linear attention (Yang et al., 2023) and slot memories (Zhang et al., 2024; Afzal et al., 2026) compress an unbounded history into a constant-size learned state, where decay by recency and routing by content are natural choices. Video breaks both assumptions, because what must be recalled depends mainly on locality. Current video adaptations of these models (Po et al., 2025; StateSpaceDiffuser) are bounded, but lack selective routing - they treat the camera only as an input, while updating the full state at every step. Yet, naively introducing content-based routing confuses visually similar places (please see our answer to Q2, where content-only keys lose up to 10.1 dB). Our contribution is to introduce selective, pose-based routing for a bounded implicit memory.
Explicit, retrieval-based memories as in WorldMem (Xiao et al., 2026), MosaicMem (Yu et al., 2026), and Lyra 2.0 (Bahmani et al., 2026) store an ever-growing history and use the camera only to select what is read. SMART keeps the bounded, learned state of implicit memory but uses the camera as its address: pose alone decides which slot each region is written to (other slots stay unchanged, so forgetting depends on where, not when); values are re-posed by the target camera; and every read returns a fused draft. Storage is therefore constant, and a memory read fuses multiple observations, rather than copying a best-matching ones. Crucially, despite maintaining strictly constant storage, SMART still outperforms these explicit retrieval methods, because memory read can fuse multiple observations, not just copy a single one.
Q2. How does SMART compare directly with camera-based retrieval such as FOV IoU, and how much of the gain comes from the routing design itself?
Direct comparison with FOV-based retrieval. The comparison to GeoCache (Table 1) implements that comparison in a strong form: following Lyra 2.0, frames are even retrieved such that they cover as many target pixels as possible after reprojection with ground-truth depth, which is a stronger criterion than frustum overlap. At 3,223 frames, SMART exceeds it with 4× less compute, and constant storage. We will better highlight this comparison in the manuscript.
Isolating the routing design. The memory-only ablations change one component at a time: dense vs. routed slots (Table 3), with vs. without SE(3) value transport (Table 3), heads and slots (Tables 3 and 5), and - new in this rebuttal - pose-only vs. joint vs. content-only addressing in the same sparse memory. Starting from our Memory Maze SMART-SM (16 slots, 98,304-float state), we changed only the write key (and the router input that follows it), keeping the architecture, the data and the 100k-step training recipe fixed: (i) pose-only keys (paper), (ii) joint keys with both pose and content based keys, (iii) content-only keys. We evaluate on 250 episodes per length:
At 3,201 frames horizon, with everything else fixed, content-only keys and joint keys degrade severely relative to pose-only keys. This isolates pose-based routing as the component behind long-horizon retention.