- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes LSM-World, which improves the long-term spatial consistency of video generation by constructing persistent three-dimensional memory in the VAE latent space. The method uses depth to back-project latent features into three-dimensional space, and then directly projects them based on the pose of the target camera to obtain conditional features. The video generation is guided by control branches, and the memory is updated after each segment is generated. Its main contribution is to avoid image rendering and repetitive VAE encoding in RGB point cloud memory reading, reducing cache overhead. The experiments demonstrated competitive generation quality and consistency, as well as efficiency advantages on WorldScore and RealEstate10K.
Critical strengths
The paper directly constructs and reads the three-dimensional spatial memory in the VAE latent space. By projecting the latent features onto the target perspective, it provides geometric conditions for video generation and avoids the additional overhead of "rendering the target perspective image and then performing VAE encoding" in the RGB point cloud method. Meanwhile, the low spatial resolution of latent reduces the number of cache points and storage requirements. The overall design is simple and can be naturally integrated into the existing video diffusion model, which has practical engineering value.
Critical weaknesses
- The method excludes dynamic objects from persistent memory, thus mainly addressing the long-term consistency of static scenes. It is unable to maintain the identity and state of dynamic objects through the proposed memory mechanism.
- The use of Qwen3-VL and SAM3 for dynamic region identification and segmentation during reasoning may significantly increase the time cost, but the paper does not explicitly report this part of the overhead and whether it is included in the end-to-end time consumption.
Questions for authors
- Is the efficiency and generation effect evaluation only for static scenarios? If so, compared with methods such as Gen3C that support dynamic scenarios, are these results sufficient to support the broader advantageous conclusions of the paper?
- What is the call frequency and actual time consumption of Qwen3-VL and SAM3 during inference? Does the end-to-end time of the report include these steps? Please provide a complete time-consuming breakdown including segmentation, depth estimation, memory reading and writing, and video generation.
6 comments
Response to Reviewer qPGj (Part 2/2)
Q1. Dynamic-scene evaluation and comparison with Gen3C
Evaluation scope. No. Our evaluation includes both static and dynamic scenes. W1 reports results on WorldScore’s dynamic partition, reconstruction quality within dynamic-object regions, and a controlled ablation of dynamic filtering (Tables 6 and 17; Appendix C.6). These evaluations assess generated videos containing dynamic objects, while the persistent memory preserves static scene structure.
Direct comparison with Gen3C. We compare Gen3C and LSM-World on the same dynamic scenes, using identical input frames, target camera trajectories, and evaluation masks. Table 18 (Appendix C.6) reports PSNR, SSIM, and LPIPS within dynamic-object regions, together with generation latency covering each method’s complete inference pipeline. Section 5.2 also summarizes the results.
| Method | Dynamic-region PSNR ↑ | Dynamic-region SSIM ↑ | Dynamic-region LPIPS ↓ | Generation latency (s/frame) ↓ |
|---|---|---|---|---|
| Gen3C | 15.17 | 0.598 | 0.389 | 2.86 |
| LSM-World | 17.06 | 0.681 | 0.333 | 1.35 |
LSM-World achieves higher PSNR and SSIM and lower LPIPS than Gen3C within dynamic-object regions. The reported latency also corresponds to a 2.12× generation speedup over Gen3C.
Q2. Call frequency and runtime breakdown
Call frequency and cost. Qwen3-VL and SAM3 are called once per generated chunk, outside the diffusion denoising loop. Their measured runtimes are 0.31 s and 0.83 s, respectively, as reported in W2 and Table 16 (Appendix C.5).
Timing scope and breakdown. The additional profiling in W2 and Table 16 (Appendix C.5) extends Figure 2(c) to include both modules, alongside depth estimation, memory read/write, and video diffusion. The complete RGB–latent runtime comparison, including initialization, memory update, and system overhead, is reported separately in Table 2.
We thank you again for your constructive questions about dynamic scenes and inference overhead. We hope the dynamic-region evaluations, Gen3C comparison, and detailed runtime measurements clarify the scope and practical cost of our memory mechanism. We welcome any further comments or suggestions.
Thank you for your detailed reply and updated manuscript. My concerns have been addressed. I find the topic interesting and the contributions substantial. As a supplementary suggestion, please consider adding video examples showing dynamic objects across multiple chunks to illustrate motion continuity.
Response to Reviewer qPGj (Part 1/2)
We thank Reviewer qPGj for the careful evaluation and constructive feedback. We appreciate your recognition of the method’s “practical engineering value”, its natural integration with existing video diffusion models, and the reduction in cache overhead and storage requirements. We have carefully considered all comments and provide our point-by-point responses below.
Memory scope. Our persistent memory preserves stable scene structure, while dynamic objects are generated using the video backbone and temporal context carried across chunks. Dynamic regions are filtered only when updating the spatial memory. They remain part of the generated videos.
Writing moving objects into the memory with fixed world coordinates can impose outdated object locations and appearances on subsequent views. Filtering avoids these stale spatial constraints. The current cache does not explicitly store persistent dynamic-object identities or evolving states across revisits. This scope is clarified in Section 4.4 and the limitations in Section 6.
Effect of dynamic filtering. The controlled ablation in Table 6 (Section 5.3) supports this design. With all other settings fixed, filtering improves both generation performance on WorldScore’s dynamic partition and spatial consistency; Appendix C.6 discusses these results.
These results show that excluding dynamic objects from the persistent latent spatial memory benefits dynamic-scene generation, while the backbone and temporal context remain responsible for modeling their motion.
Dynamic-region evaluation. We further evaluate reconstruction quality within dynamic-object regions, which are excluded from the persistent spatial memory. Table 17 (Appendix C.6) reports these scores alongside full-frame results.
Call frequency. Following Spatia, we precompute dynamic-object masks offline during dataset preparation for training. During inference, Qwen3-VL and SAM3 are called once per generated chunk to obtain masks for memory updates, outside the diffusion denoising loop. Section 4.4 and Appendices C.5 and D now clarify this distinction.
Timing scope and overhead. Table 16 (Appendix C.5) extends the profiling in Figure 2(c) to include dynamic-region identification and segmentation. It reports the listed component times, with percentages computed over their sum (41.65 s). Table 2 separately reports the complete RGB–latent runtime comparison, including initialization, memory update, and system overhead.
Qwen3-VL and SAM3 together add 1.14 s, accounting for 2.74% of the measured total. Video diffusion remains the dominant cost at 94.60%.
(see next comments for more response, thanks)