- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes L3G-World, an autoregressive world generation framework that uses a latent 3D Gaussian memory for long-horizon spatial consistency. A feed-forward updater (L3G-Mem, built on ZipSplat with a DA3 backbone) maintains the memory. It spawns new Gaussians only in regions where the rendered opacity is low or the RGB error is high, and it refines visible Gaussians through gated residual updates, so the memory grows with explored scene extent rather than stream length. Each Gaussian also stores a VAE latent feature. This lets the memory be rendered directly into the latent space of the video DiT (Cosmos-Predict2.5) and channel-concatenated as conditioning, which removes the RGB-render-then-VAE-encode step. The generator is distilled into a few-step causal model with AnyFlow. The paper reports 9 FPS generation, along with large reductions in primitive count, reconstruction time, and conditioning latency compared with prior explicit 3D-memory methods.
Critical strengths
-
Storing VAE latents directly on the Gaussians is a neat idea. It avoids the render-then-VAE-encode step that previous 3D-memory methods need, and the ablation shows it matters for both speed and quality.
-
The memory is much more compact and faster than prior explicit 3D memories, which makes it practical to plug into a real-time generator.
-
Loop-closure consistency is strong, and the refinement ablation supports the point that memory should be updated rather than only accumulated.
Critical weaknesses
-
Limited novelty. Rendering a persistent 3D memory into target views to condition video generation is already established (ViewCrafter, GEN3C, Lyra), and VMem already maintains an updated, merged 3D memory instead of an accumulated one. Compared with these, the contribution here is mainly a more compact memory with VAE latents stored on the Gaussians. That is a good efficiency gain, but it is not a new idea. The real-time part also comes from existing distillation methods.
-
Explicit 3D memory has a fundamental problem. The memory commits to metric depth from DA3 at initialization, and far-field depth is essentially unobservable. Once the depth is wrong, the generator follows it, and the generated frames confirm the wrong geometry during refinement, so the error can never be corrected. This should cause distortion of distant content, and geometry that falls apart when the camera moves forward. The current benchmarks, which mostly contain near-field scenes and lateral trajectories, would not reveal this. I see this as a limitation of the paradigm itself, not something a better updater can fix.
Questions for authors
1.How much of the loop-closure consistency comes from the 3D memory itself, and how much from the pose-based reference retrieval? An ablation that removes each one separately would help.
2.Beyond efficiency, what can this memory do that GEN3C-style or VMem-style memories cannot? Is there any scenario in which it improves consistency or quality for reasons other than speed?
3.How does the method behave in scenes with a lot of far-field content, or when the camera keeps moving forward toward distant objects? Can refinement recover from wrong initial depth?
8 comments
Response to reviewer GjYJ (Part 2)
W1. Clarification of Methodological Novelty
We agree that rendering 3D memory to condition video generation is already established. We would like to clarify two complementary contributions of L3G-World: (1) a methodological contribution in correcting geometric errors in 3D memory through autoregressive updates, and (2) a system-level contribution in extending compact Gaussians into persistent memory and integrating efficient memory updates and conditioning into an end-to-end real-time world generation framework. The latter was also recognized by reviewer pvC9, who described our work as "a strong systems combination for world-model memory." Importantly, our key distinction from existing 3D memory-based world models, including VMem [1], lies in explicitly refining previously reconstructed geometry to correct geometric errors.
This distinction is particularly important for autoregressive world generation. Initial 3D reconstruction may contain geometric errors, while subsequently generated video chunks provide additional viewpoints that can reveal these errors. Thus, the 3D memory should leverage new video frames to refine existing geometry, rather than merely accumulating or filtering incoming primitives. However, existing models such as ViewCrafter [2], GEN3C [3], and Lyra 2.0 [4] use simple accumulation of 3D primitives without refinement. Although VMem [1] filters out incoming primitives that overlap with existing ones, it does not refine the previously reconstructed geometry itself. Thus, geometric errors introduced during previous reconstruction remain in memory and persist through subsequent generation.
In contrast, L3G-Mem introduces two key mechanisms for autoregressive 3D memory updates. (1) Gaussian refinement retrieves existing Gaussians and corrects their geometric errors through residual updates using newly generated video chunks. This allows previously reconstructed geometry to be progressively corrected rather than remaining fixed throughout subsequent generation. (2) Adaptive spawning selectively creates new Gaussians in insufficiently represented regions, with the number of spawned Gaussians adaptively determined by the extent of these regions. This reduces redundant accumulation and allows memory to grow primarily with explored scene extent rather than generation length.
In addition to these methodological contributions, we would also like to highlight the system-level contribution of enabling end-to-end real-time world generation with persistent 3D memory. Rather than simply adopting compact Gaussians for memory, L3G-World extends them into persistent, autoregressively updated world-model memory. Although few-step video distillation enables real-time video generation, it alone does not guarantee real-time performance when 3D memory updates and conditioning introduce substantial overhead. To address these bottlenecks, L3G-World combines efficient feed-forward updates of compact Gaussians through adaptive spawning and refinement with low-latency conditioning through VAE latent lifting and direct rendering. This integration enables real-time autoregressive world generation while maintaining persistent 3D memory for long-horizon spatial consistency and geometric error correction.
Thus, L3G-World contributes both a novel autoregressive memory update mechanism that corrects previously reconstructed geometry and an effective system-level framework that integrates persistent 3D memory into real-time world generation.
We will clearly describe our key contributions in the camera-ready version.
References
[1] Li et al. "VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory." ICCV, 2025.
[2] Yu et al. "ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis." TPAMI, 2025.
[3] Ren et al. "GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control." CVPR, 2025.
[4] Shen et al. "Lyra 2.0: Explorable Generative 3D Worlds." SIGGRAPH Asia, 2026.
Response to reviewer GjYJ (Part 3)
W2 & Q3. Robustness to Geometric Errors and Far-Field Depth Inaccuracies
We have included additional qualitative results in Fig. 10 of the revised manuscript to demonstrate that L3G-Gen can generate plausible videos despite geometric errors in the 3D memory, while L3G-Mem can progressively correct these errors using newly generated video frames.
We agree that initial far-field depth estimates can be inaccurate, potentially causing geometric distortions as the camera moves forward. However, L3G-Gen leverages spatially aligned information from the 3D memory together with its learned spatiotemporal generative prior. As an image-to-video diffusion model, L3G-Gen learns how scene appearance evolves across frames under camera motion. Given an initial image and a camera trajectory, this prior enables the model to generate plausible and temporally coherent views, even when the memory renderings contain geometric distortions. The rendered 3D memory serves as spatially aligned guidance rather than a strict geometric constraint. Thus, accurate memory regions help preserve scene structure and appearance, while the learned prior allows L3G-Gen to generate plausible content without strictly reproducing rendering artifacts caused by inaccurate geometry, including in far-field regions.
As shown in Fig. 10, we provide three examples:
- Top: The pathway appears distorted in the initial memory rendering due to depth inaccuracies, whereas L3G-Gen generates a plausible pathway. The subsequently updated memory reconstructs the pathway more accurately.
- Middle: A distant tree becomes distorted in the memory rendering as the camera moves forward, while L3G-Gen generates the tree without noticeable distortion. L3G-Mem then refines the 3D memory, removing duplicated tree structures.
- Bottom: As the camera moves forward through a narrow passage between buildings, initial depth errors produce floating Gaussians and blurry memory renderings. Nevertheless, L3G-Gen generates a coherent video, allowing L3G-Mem to remove the floating Gaussians and reconstruct the passage more faithfully.
These examples demonstrate that geometric errors in the 3D memory do not necessarily propagate to subsequent video generation. By combining spatial information from the 3D memory with its learned spatiotemporal generative prior, L3G-Gen can generate plausible video chunks even when the memory contains geometric inaccuracies. L3G-Mem then uses these newly generated frames to refine existing Gaussians through residual updates, progressively correcting geometric errors in the persistent 3D memory. This autoregressive interaction between video generation and memory refinement allows previously reconstructed geometry to improve over time, mitigating the effects of inaccurate initial depth estimates.
We have included further analyses in Sec. D of the revised manuscript, including visualizations of 3D memory updates and conditioning (Fig. 8) and an overview of the autoregressive pipeline (Fig. 9).
Q1. Ablations on 3D Memory and History-Frame Retrieval
| Variant | Self-PSNR | Self-SSIM | Self-LPIPS | LPIPS | FID | IQ | AQ | SC | BC | TransErr | RotErr | MEt3R |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| w/o 3D Gaussian Memory | 13.31 | 0.389 | 0.629 | 0.699 | 64.05 | 0.631 | 0.484 | 0.785 | 0.780 | 0.849 | 8.64 | 0.076 |
| w/o History Frame Retrieval | 13.53 | 0.399 | 0.597 | 0.636 | 64.35 | 0.588 | 0.460 | 0.775 | 0.870 | 0.719 | 11.99 | 0.087 |
| L3G-World (Full) | 15.23 | 0.423 | 0.447 | 0.583 | 45.74 | 0.653 | 0.516 | 0.811 | 0.887 | 0.249 | 5.98 | 0.082 |
We provide an additional ablation study isolating the contributions of 3D memory and history-frame retrieval. Removing 3D memory yields the worst self-consistency and the largest degradation in translation accuracy, while also degrading generation quality. This highlights the importance of spatially aligned memory conditioning for scene consistency, camera controllability, and generation quality. Disabling history-frame retrieval also degrades self-consistency and overall generation quality, demonstrating that both components contribute to maintaining consistent, high-quality world generation. We have included these results in Tab. 4 of the revised manuscript.
Response to reviewer GjYJ (Part 1)
We sincerely thank the reviewer for the careful and constructive evaluation of our work. We appreciate your recognition of the benefits of direct VAE latent rendering, substantial memory efficiency improvements, and the effectiveness of memory refinement.
Other reviewers also highlighted the following strengths:
Your questions regarding methodological novelty, robustness to geometric errors, and the individual contributions of 3D memory and history-frame retrieval raise important points. We have conducted additional experiments and analyses, and hope our responses address these concerns. For clarity, we have highlighted the revisions addressing your comments in purple in the revised manuscript.