acceptodds
Under review as a conference paper at ICLR 2027

L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

Abstract

Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either frame retrieval or explicit 3D memories, both of which scale poorly as memory grows with the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which adds computation and propagates rendering artifacts. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose \ours, in which a memory updater and a video diffusion transformer alternate autoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones in explored regions, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores a VAE latent feature, allowing the memory to be rendered directly into the DiT's latent space without VAE re-encoding. Experiments show that \ours achieves state-of-the-art generation quality and long-horizon spatial consistency, with up to fewer 3D primitives, faster memory update, and faster memory rendering than prior 3D memory-based methods, enabling real-time world generation at 9 FPS.

Primary area:
generative models
Keywords:
Real Time Long Sequence Video Generation, World Generation, Video Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GjYJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes L3G-World, an autoregressive world generation framework that uses a latent 3D Gaussian memory for long-horizon spatial consistency. A feed-forward updater (L3G-Mem, built on ZipSplat with a DA3 backbone) maintains the memory. It spawns new Gaussians only in regions where the rendered opacity is low or the RGB error is high, and it refines visible Gaussians through gated residual updates, so the memory grows with explored scene extent rather than stream length. Each Gaussian also stores a VAE latent feature. This lets the memory be rendered directly into the latent space of the video DiT (Cosmos-Predict2.5) and channel-concatenated as conditioning, which removes the RGB-render-then-VAE-encode step. The generator is distilled into a few-step causal model with AnyFlow. The paper reports 9 FPS generation, along with large reductions in primitive count, reconstruction time, and conditioning latency compared with prior explicit 3D-memory methods.

Critical strengths
  1. Storing VAE latents directly on the Gaussians is a neat idea. It avoids the render-then-VAE-encode step that previous 3D-memory methods need, and the ablation shows it matters for both speed and quality.

  2. The memory is much more compact and faster than prior explicit 3D memories, which makes it practical to plug into a real-time generator.

  3. Loop-closure consistency is strong, and the refinement ablation supports the point that memory should be updated rather than only accumulated.

Critical weaknesses
  1. Limited novelty. Rendering a persistent 3D memory into target views to condition video generation is already established (ViewCrafter, GEN3C, Lyra), and VMem already maintains an updated, merged 3D memory instead of an accumulated one. Compared with these, the contribution here is mainly a more compact memory with VAE latents stored on the Gaussians. That is a good efficiency gain, but it is not a new idea. The real-time part also comes from existing distillation methods.

  2. Explicit 3D memory has a fundamental problem. The memory commits to metric depth from DA3 at initialization, and far-field depth is essentially unobservable. Once the depth is wrong, the generator follows it, and the generated frames confirm the wrong geometry during refinement, so the error can never be corrected. This should cause distortion of distant content, and geometry that falls apart when the camera moves forward. The current benchmarks, which mostly contain near-field scenes and lateral trajectories, would not reveal this. I see this as a limitation of the paradigm itself, not something a better updater can fix.

Questions for authors

1.How much of the loop-closure consistency comes from the 3D memory itself, and how much from the pose-based reference retrieval? An ablation that removes each one separately would help.

2.Beyond efficiency, what can this memory do that GEN3C-style or VMem-style memories cannot? Is there any scenario in which it improves consistency or quality for reasons other than speed?

3.How does the method behave in scenes with a lot of far-field content, or when the camera keeps moving forward toward distant objects? Can refinement recover from wrong initial depth?

8 comments
AuthorsOpenReview ↗

Response to reviewer GjYJ (Part 1)

We sincerely thank the reviewer for the careful and constructive evaluation of our work. We appreciate your recognition of the benefits of direct VAE latent rendering, substantial memory efficiency improvements, and the effectiveness of memory refinement.

Other reviewers also highlighted the following strengths:

  • The importance and timely motivation of addressing memory scalability and long-horizon spatial consistency in real-time world generation (pvC9).
  • The strong system-level contribution of effectively integrating components into a practical world-model memory framework (pvC9).
  • The comprehensive experimental evaluation, including comparisons with both world generation and feed-forward 3D reconstruction methods (MPUu).

Your questions regarding methodological novelty, robustness to geometric errors, and the individual contributions of 3D memory and history-frame retrieval raise important points. We have conducted additional experiments and analyses, and hope our responses address these concerns. For clarity, we have highlighted the revisions addressing your comments in purple in the revised manuscript.

AuthorsOpenReview ↗

Response to reviewer GjYJ (Part 2)

W1. Clarification of Methodological Novelty

We agree that rendering 3D memory to condition video generation is already established. We would like to clarify two complementary contributions of L3G-World: (1) a methodological contribution in correcting geometric errors in 3D memory through autoregressive updates, and (2) a system-level contribution in extending compact Gaussians into persistent memory and integrating efficient memory updates and conditioning into an end-to-end real-time world generation framework. The latter was also recognized by reviewer pvC9, who described our work as "a strong systems combination for world-model memory." Importantly, our key distinction from existing 3D memory-based world models, including VMem [1], lies in explicitly refining previously reconstructed geometry to correct geometric errors.

This distinction is particularly important for autoregressive world generation. Initial 3D reconstruction may contain geometric errors, while subsequently generated video chunks provide additional viewpoints that can reveal these errors. Thus, the 3D memory should leverage new video frames to refine existing geometry, rather than merely accumulating or filtering incoming primitives. However, existing models such as ViewCrafter [2], GEN3C [3], and Lyra 2.0 [4] use simple accumulation of 3D primitives without refinement. Although VMem [1] filters out incoming primitives that overlap with existing ones, it does not refine the previously reconstructed geometry itself. Thus, geometric errors introduced during previous reconstruction remain in memory and persist through subsequent generation.

In contrast, L3G-Mem introduces two key mechanisms for autoregressive 3D memory updates. (1) Gaussian refinement retrieves existing Gaussians and corrects their geometric errors through residual updates using newly generated video chunks. This allows previously reconstructed geometry to be progressively corrected rather than remaining fixed throughout subsequent generation. (2) Adaptive spawning selectively creates new Gaussians in insufficiently represented regions, with the number of spawned Gaussians adaptively determined by the extent of these regions. This reduces redundant accumulation and allows memory to grow primarily with explored scene extent rather than generation length.

In addition to these methodological contributions, we would also like to highlight the system-level contribution of enabling end-to-end real-time world generation with persistent 3D memory. Rather than simply adopting compact Gaussians for memory, L3G-World extends them into persistent, autoregressively updated world-model memory. Although few-step video distillation enables real-time video generation, it alone does not guarantee real-time performance when 3D memory updates and conditioning introduce substantial overhead. To address these bottlenecks, L3G-World combines efficient feed-forward updates of compact Gaussians through adaptive spawning and refinement with low-latency conditioning through VAE latent lifting and direct rendering. This integration enables real-time autoregressive world generation while maintaining persistent 3D memory for long-horizon spatial consistency and geometric error correction.

Thus, L3G-World contributes both a novel autoregressive memory update mechanism that corrects previously reconstructed geometry and an effective system-level framework that integrates persistent 3D memory into real-time world generation.

We will clearly describe our key contributions in the camera-ready version.

References

[1] Li et al. "VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory." ICCV, 2025.

[2] Yu et al. "ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis." TPAMI, 2025.

[3] Ren et al. "GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control." CVPR, 2025.

[4] Shen et al. "Lyra 2.0: Explorable Generative 3D Worlds." SIGGRAPH Asia, 2026.

AuthorsOpenReview ↗

Response to reviewer GjYJ (Part 3)

W2 & Q3. Robustness to Geometric Errors and Far-Field Depth Inaccuracies

We have included additional qualitative results in Fig. 10 of the revised manuscript to demonstrate that L3G-Gen can generate plausible videos despite geometric errors in the 3D memory, while L3G-Mem can progressively correct these errors using newly generated video frames.

We agree that initial far-field depth estimates can be inaccurate, potentially causing geometric distortions as the camera moves forward. However, L3G-Gen leverages spatially aligned information from the 3D memory together with its learned spatiotemporal generative prior. As an image-to-video diffusion model, L3G-Gen learns how scene appearance evolves across frames under camera motion. Given an initial image and a camera trajectory, this prior enables the model to generate plausible and temporally coherent views, even when the memory renderings contain geometric distortions. The rendered 3D memory serves as spatially aligned guidance rather than a strict geometric constraint. Thus, accurate memory regions help preserve scene structure and appearance, while the learned prior allows L3G-Gen to generate plausible content without strictly reproducing rendering artifacts caused by inaccurate geometry, including in far-field regions.

As shown in Fig. 10, we provide three examples:

  • Top: The pathway appears distorted in the initial memory rendering due to depth inaccuracies, whereas L3G-Gen generates a plausible pathway. The subsequently updated memory reconstructs the pathway more accurately.
  • Middle: A distant tree becomes distorted in the memory rendering as the camera moves forward, while L3G-Gen generates the tree without noticeable distortion. L3G-Mem then refines the 3D memory, removing duplicated tree structures.
  • Bottom: As the camera moves forward through a narrow passage between buildings, initial depth errors produce floating Gaussians and blurry memory renderings. Nevertheless, L3G-Gen generates a coherent video, allowing L3G-Mem to remove the floating Gaussians and reconstruct the passage more faithfully.

These examples demonstrate that geometric errors in the 3D memory do not necessarily propagate to subsequent video generation. By combining spatial information from the 3D memory with its learned spatiotemporal generative prior, L3G-Gen can generate plausible video chunks even when the memory contains geometric inaccuracies. L3G-Mem then uses these newly generated frames to refine existing Gaussians through residual updates, progressively correcting geometric errors in the persistent 3D memory. This autoregressive interaction between video generation and memory refinement allows previously reconstructed geometry to improve over time, mitigating the effects of inaccurate initial depth estimates.

We have included further analyses in Sec. D of the revised manuscript, including visualizations of 3D memory updates and conditioning (Fig. 8) and an overview of the autoregressive pipeline (Fig. 9).


Q1. Ablations on 3D Memory and History-Frame Retrieval

VariantSelf-PSNR ↑\uparrowSelf-SSIM ↑\uparrowSelf-LPIPS ↓\downarrowLPIPS ↓\downarrowFID ↓\downarrowIQ ↑\uparrowAQ ↑\uparrowSC ↑\uparrowBC ↑\uparrowTransErr ↓\downarrowRotErr ↓\downarrowMEt3R ↓\downarrow
w/o 3D Gaussian Memory13.310.3890.6290.69964.050.6310.4840.7850.7800.8498.640.076
w/o History Frame Retrieval13.530.3990.5970.63664.350.5880.4600.7750.8700.71911.990.087
L3G-World (Full)15.230.4230.4470.58345.740.6530.5160.8110.8870.2495.980.082

We provide an additional ablation study isolating the contributions of 3D memory and history-frame retrieval. Removing 3D memory yields the worst self-consistency and the largest degradation in translation accuracy, while also degrading generation quality. This highlights the importance of spatially aligned memory conditioning for scene consistency, camera controllability, and generation quality. Disabling history-frame retrieval also degrades self-consistency and overall generation quality, demonstrating that both components contribute to maintaining consistent, high-quality world generation. We have included these results in Tab. 4 of the revised manuscript.

Reviewer_pvC9OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper targets real-time, long-horizon interactive world generation with spatial consistency. Existing methods either grow temporal KV/frame memory with stream length, or use explicit 3D memories that either accumulate dense primitives (O(TR)) or require slow optimization. The authors propose L3G-World with a Latent 3D Gaussian Memory: L3G-Mem adaptively spawns Gaussians only for missing/high-error regions and refines existing ones, so memory scales mainly with scene extent; each Gaussian stores a VAE latent, enabling direct latent-space rendering for DiT conditioning without RGB→VAE re-encoding. L3G-Gen is a memory-conditioned video DiT, further distilled for few-step inference. Experiments on DL3DV and RealEstate10K report improved loop-closure self-consistency, large gains in primitives/memory/rendering speed (up to hundreds× VAE rendering), and competitive generation quality, with ablations supporting adaptive spawning, refinement, and VAE lifting.

Critical strengths
  1. The problem framing is clear and timely: long-horizon consistency vs. real-time latency under growing 3D memory.

Two design choices are well motivated and complementary: selective spawn+refine keeps memory compact and correctable; VAE-latent Gaussians remove the RGB render to VAE bottleneck.

  1. Efficiency results are substantial and carefully reported (primitives, memory, recon latency, RGB vs VAE FPS) against strong explicit-memory baselines.

  2. Ablations are informative: adaptive spawning reduces redundancy, refinement helps long-horizon consistency/pose, and VAE lifting is critical for latent conditioning speed/quality.

  3. Loop-closure evaluation is a sensible probe of spatial memory, and qualitative results support reduced drift/artifacts vs. several baselines.

Critical weaknesses
  1. “SOTA generation quality” is overclaimed. On DL3DV, L3G-World has best PSNR/LPIPS for self-consistency, but SSIM is behind SpMem/3DScenePrompt; pose and MEt3R are not best. Distilled RealEstate10K quality drops sharply (PSNR 19.42→15.82, LPIPS 0.219→0.522). The paper should report a clearer win/loss breakdown instead of a blanket SOTA quality statement.

  2. Abstract/Table 1 emphasize ~9 FPS real-time; main efficiency comparison says baselines are measured on one A100, while the 9 FPS figure is on two H100s. Please report end-to-end FPS for all methods under the same hardware, or clearly separate “memory subsystem speed” from “system FPS.” Resolutions also differ across methods in Table 1.

  3. Evaluation scope is narrower than the “infinite interactive world” claim. Results are mainly on static-scene exploration trajectories from DL3DV/RE10K with scripted loop closure. There is limited analysis of highly dynamic scenes, free user interaction, failure accumulation over much longer horizons, or open-world diversity beyond these datasets.

  4. Novelty is partly compositional. The system builds heavily on ZipSplat, Depth Anything 3, Cosmos-Predict2.5, and AnyFlow. The contribution is a strong systems combination for world-model memory, but the paper should more sharply isolate what is new versus inherited components

Questions for authors

Please see weaknesses.

5 comments
AuthorsOpenReview ↗

Response to reviewer pvC9 (Part 1)

We sincerely thank the reviewer for the detailed and constructive feedback. We appreciate your recognition of our well-motivated memory design, strong system-level contribution, substantial efficiency gains, and informative ablations and loop-closure evaluation.

Other reviewers also highlighted the following strengths:

  • The technical soundness and intuitive design of directly storing and alpha-compositing VAE latent features on 3D Gaussians (MPUu).
  • The practical benefits of compact and efficient 3D memory for integration into real-time world generation (GjYJ).

Your comments on the scope of our claims, evaluation settings, and methodological novelty provide valuable guidance. We have prepared additional results and clarifications to address these concerns, as detailed below. For clarity, we have highlighted the revisions addressing your comments in blue in the revised manuscript.


W1. Scope of SOTA Claims

We agree that the term "SOTA generation quality" is too broad. We have revised the Abstract and Introduction to more precisely highlight our strengths in long-horizon self-consistency (particularly PSNR and LPIPS) and memory efficiency, while describing generation quality as competitive rather than uniformly state-of-the-art.

Specifically, on DL3DV, L3G-World and its distilled variant achieve the best and second-best self-consistency PSNR and LPIPS, while demonstrating strong generation quality. On RealEstate10K, L3G-World achieves the best self-consistency PSNR and LPIPS, along with the best generation LPIPS and FID, and a tied-best MEt3R score.


W2. Hardware Settings and FPS Comparisons

We apologize for the lack of clarity regarding the hardware settings. We have revised the manuscript to clearly distinguish end-to-end generation FPS from 3D memory efficiency.

  • Tab. 1: All end-to-end generation FPS and latency measurements, including those of our method and the baselines, were obtained on a single H100 GPU.
  • Tab. 2: All memory efficiency metrics were measured on a single A100 GPU for both our method and the baselines.

Thus, all methods within each table were evaluated under identical hardware settings, ensuring hardware-matched comparisons.

We also acknowledge an error in the Introduction, where we incorrectly stated that the 9.01 FPS result was measured on two H100 GPUs. The correct configuration is a single H100 GPU. We have corrected this error and clarified the hardware settings in the captions of Tabs. 1 and 2, as well as throughout the revised manuscript, including the Introduction and Sec. 4.1. Regarding resolution differences, Tab. 1 reports each method at its native resolution to avoid evaluating baselines at resolutions outside their original operating settings.


W3. Additional Evidence of Interactive World Generation

To further demonstrate the interactive capabilities and generalization of L3G-World, we have included additional qualitative results in Figs. 3, 5, and 6 of the revised manuscript.

Specifically, Fig. 3(b) demonstrates interactive long-horizon generation following user-controlled camera trajectories via keyboard inputs while maintaining scene consistency. Fig. 3(c) shows world generation along diverse camera trajectories, and Fig. 3(d) demonstrates generalization across various open-world scenes. Figs. 5 and 6 provide further qualitative results across a wider range of camera trajectories and open-world scenes.

In addition, we would like to clarify that L3G-World is primarily designed for static scenes, as discussed in Sec. E (Sec. F in the revised manuscript).

AuthorsOpenReview ↗

Response to reviewer pvC9 (Part 2)

W4. Clarification of Methodological Novelty

We would like to clarify which components are newly introduced in L3G-World, rather than inherited from the adopted models. Specifically, (1) Gaussian refinement, (2) adaptive spawning, and (3) VAE latent lifting are novel mechanisms that are not provided by the adopted models, such as ZipSplat or Depth Anything 3. Together, these mechanisms extend feed-forward reconstruction into an autoregressive 3D memory framework for world models, enabling continuous geometric error correction, adaptive memory expansion, and efficient conditioning for subsequent video generation.

Feed-forward reconstruction and geometry estimation models [1–6], such as Depth Anything 3 [1] and ZipSplat [2], are designed for scene reconstruction, rather than maintaining and refining persistent 3D representations across video chunks. Nevertheless, existing 3D memory-based world models [7–11] often employ off-the-shelf reconstruction models, reconstructing newly generated video chunks into 3D primitives and incorporating them into persistent memory without refining previously stored geometry. Consequently, accumulating these primitives can introduce redundancy in previously explored regions, causing memory consumption to grow with generation length and potentially producing rendering artifacts. Moreover, geometric errors in the stored 3D memory may remain uncorrected throughout subsequent generation. Furthermore, rendering 3D memory into RGB images from target viewpoints and re-encoding them into VAE latents can introduce substantial overhead and latency, preventing the entire autoregressive pipeline from operating in real time.

To overcome these limitations, we introduce a novel autoregressive 3D memory framework tailored to real-time, spatially consistent world generation. Specifically, our three proposed mechanisms operate as follows: (1) Gaussian refinement retrieves Gaussians from the current memory and corrects their geometric errors through residual updates using newly generated video chunks. (2) Adaptive spawning selectively creates new Gaussians in insufficiently represented regions, with the number of spawned Gaussians adaptively determined by the extent of these regions. This reduces redundant accumulation and allows memory to grow primarily with explored scene extent rather than generation length. (3) VAE latent lifting stores VAE latent features in each Gaussian, enabling direct rendering into the generator's latent space without additional VAE encoding.

Together, these mechanisms enable a 3D memory framework that supports scalable memory growth and geometric error correction through efficient feed-forward updates, while providing low-latency conditioning through direct VAE rendering. Thus, our contribution lies in designing an autoregressive 3D memory framework specifically tailored to the efficiency and spatial consistency requirements of real-time world generation, rather than merely integrating existing components.

In the revised manuscript, we have included Fig. 8(a) to visualize our autoregressive 3D memory update, as well as Figs. 3(a) and 10 to demonstrate how geometric errors in the 3D memory are corrected using newly generated video chunks.

We will further clarify our key contributions in the camera-ready version.

References

[1] Lin et al. "Depth Anything 3: Recovering the Visual Space from Any Views." ICLR, 2026.

[2] Veicht et al. "ZipSplat: Fewer Gaussians, Better Splats." arXiv:2606.05102, 2026.

[3] Wang et al. "MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision." CVPR, 2025.

[4] Wang et al. "VGGT: Visual Geometry Grounded Transformer." CVPR, 2025.

[5] Keetha et al. "MapAnything: Universal Feed-Forward Metric 3D Reconstruction." 3DV, 2026.

[6] Huang et al. "ViPE: Video Pose Engine for 3D Geometric Perception." arXiv:2508.10934, 2025.

[7] Zhao et al. "Spatia: Video Generation with Updatable Spatial Memory." CVPR, 2026.

[8] Ren et al. "GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control." CVPR, 2025.

[9] Lee et al. "3D Scene Prompting for Scene-Consistent Camera-Controllable Video Generation." ICLR, 2026.

[10] Yang et al. "NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos." CVPR, 2026.

[11] Shen et al. "Lyra 2.0: Explorable Generative 3D Worlds." SIGGRAPH Asia, 2026.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal – Reviewer pvC9

Dear Reviewer pvC9,

Thank you again for your detailed and constructive review. As the discussion deadline approaches, we would like to briefly follow up on our rebuttal.

To address your concerns, we have made the following revisions and clarifications:

  • W1: Refined our SOTA claims to better reflect the experimental results.
  • W2: Clarified the hardware settings and resolution differences.
  • W3: Added qualitative results on interactive generation and generalization (Figs. 3, 5, and 6).
  • W4: Clarified our methodological novelty and distinctions from existing components or approaches.

We would appreciate it if you could review our responses and share whether they address your concerns. We would be happy to clarify any remaining questions before the deadline.

Thank you again for your time and consideration.

Reviewer_MPUuOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

L3G-World is composed of two parts: a feed-forward 3D Gaussian representation, the L3G-Mem acting as the memory, and a distilled video diffusion transformer, the L3G-Gen. The key contribution is they store VAE latents directly on gaussians which naturally fits the alpha-compositing rendering. Combined with adaptive gaussian spawning, this can significantly speed up the memory conditioning for DiT, achieving real-time generation.

Critical strengths
  1. the majority of the paper's contribution is focused on efficiency, and what was proposed is a sound and reasonable approach. Storing and alpha-compositing 16-channel VAE latents directly on 3D Gaussians bypasses RGB rendering + VAE encoding, while selective spawning + residual refinement keeps primitive counts bounded to avoid memory growth issue.
  2. the suite of benchmarks, and ablations are quite comprehensive, including comparisons with both world modeling baselines and feed forward 3D Gaussian based reconstruction methods. Each core component has results on ablation and extensive analysis like loop-closure was also evaluated in addition to the main metrics.
Critical weaknesses

None

Questions for authors

I was slightly confused when I was looking at the exact setup on hardware used that produced the efficiency measures. It seems both H100-based experiments and A100 based were discussed here, which one did the main FPS measurement use?

3 comments
AuthorsOpenReview ↗

Response to reviewer MPUu

We sincerely thank the reviewer for the positive and thoughtful assessment of our work. We appreciate your recognition of the technical soundness of our approach, particularly the efficiency gains from direct VAE latent rendering and adaptive memory updates, as well as our comprehensive experimental evaluation.

Other reviewers also highlighted the following strengths:

  • The clear and timely motivation of balancing long-horizon spatial consistency with real-time efficiency (pvC9).
  • Strong loop-closure consistency and the effectiveness of memory refinement in maintaining scene consistency (GjYJ).

Your question helped us identify an ambiguity in our hardware reporting. We clarify the experimental settings below and hope this resolves the ambiguity.


Q1. Clarification of Hardware Settings for FPS Evaluation

We apologize for the lack of clarity regarding the hardware settings.

  • Tab. 1: All end-to-end generation FPS and latency measurements, including those of our method and the baselines, were obtained on a single H100 GPU.
  • Tab. 2: All memory efficiency metrics were measured on a single A100 GPU for both our method and the baselines.

Thus, all methods within each table were evaluated under identical hardware settings, ensuring hardware-matched comparisons.

We also acknowledge an error in the Introduction, where we incorrectly stated that the 9.01 FPS result was measured on two H100 GPUs. The correct configuration is a single H100 GPU. We have corrected this error and explicitly clarified the hardware settings in the captions of Tabs. 1 and 2, as well as throughout the revised manuscript, including the Introduction and Sec. 4.1.

I appreciate the clarification, and I will stand on my initial clear acceptance rating.

AuthorsOpenReview ↗

Response to reviewer MPUu

We sincerely thank you for your positive assessment and continued support. We also greatly appreciate your thoughtful feedback and consideration throughout the review process.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal and Revision Summary

Dear AC and Reviewers,

We sincerely thank the reviewers for their thoughtful and constructive feedback throughout the discussion. We are encouraged by their recognition of the technical soundness of our approach, strong long-horizon spatial consistency, substantial memory efficiency improvements, and system-level contributions to real-time world generation. Their feedback has helped us improve the clarity and experimental validation of our work. We have conducted additional experiments and analyses and updated the manuscript and supplementary material accordingly. We summarize the major updates and discussion outcomes below.


Main Paper & Appendix

  • Generation quality claims: We have revised the Abstract and Introduction to emphasize our strengths in long-horizon self-consistency and memory efficiency while avoiding overly broad SOTA generation quality claims. (Reviewer pvC9, W1)
  • Hardware settings and efficiency evaluation: We have clarified the hardware settings in Tabs. 1 and 2 and the manuscript. All end-to-end generation FPS measurements were obtained on a single H100 GPU, while 3D memory efficiency metrics were measured on a single A100 GPU. We also clarified the differences in native resolutions. (Reviewers MPUu, Q1; pvC9, W2)
  • Interactive generation and generalization: We have expanded Fig. 3 to demonstrate interactive generation with keyboard-controlled camera trajectories, diverse camera motions, and generalization across open-world scenes. We have also added Figs. 5 and 6 with further qualitative results. (Reviewer pvC9, W3)
  • Geometric error correction: We have added Fig. 10 to demonstrate how L3G-Gen generates plausible videos despite geometric inaccuracies in the 3D memory, while L3G-Mem progressively refines the memory using newly generated video frames. (Reviewer GjYJ, W2 & Q3)
  • Component ablations: We have added an ablation study in Tab. 4 to isolate the contributions of explicit 3D memory and history-frame retrieval to self-consistency, generation quality, and camera controllability. (Reviewer GjYJ, Q1)
  • Additional memory analyses: We have added Figs. 8 and 9 in Appendix D to visualize 3D memory updates, memory conditioning, and the autoregressive world generation pipeline.

Additional Far-Field Experiments

Following reviewer GjYJ's request for longer forward-motion trajectories, we provided three additional qualitative videos in the revised supplementary material (rebuttal_GjYJ_far_field_01.mp4–03.mp4). These examples demonstrate long-range camera motion toward initially distant buildings over multiple autoregressive chunks.

The videos illustrate how L3G-Mem progressively refines initially sparse far-field geometry into a more detailed representation aligned with newly generated video frames, while L3G-Gen maintains plausible video generation despite geometric inaccuracies in the memory.


Methodological Clarifications

In our individual responses, we have clarified our proposed mechanisms and their distinctions from existing approaches, particularly Gaussian refinement for correcting previously stored geometry, as opposed to primitive accumulation or redundancy filtering. We have also highlighted the system-level contribution of integrating efficient memory updates and direct VAE latent conditioning into an end-to-end real-time world generation framework. (Reviewers pvC9, W4; GjYJ, W1)

Regarding evaluation scope, we have clarified that interactive generation refers to real-time video generation following user-controlled camera trajectories, while acknowledging that our current framework primarily focuses on static scenes. We will further clarify our key methodological contributions in the camera-ready manuscript.


Reviewer Follow-up & Discussion Outcomes

We sincerely appreciate the reviewers' continued engagement and their consideration of our additional evidence and clarifications.

Reviewer MPUu reaffirmed their initial Clear Accept assessment after our hardware clarification. Reviewer pvC9 confirmed that our revised generation-quality claims and hardware clarifications adequately addressed W1 and W2, and acknowledged the improvements in experimental clarity and the value of our memory-design contributions, while maintaining reservations regarding evaluation scope and methodological novelty.

Reviewer GjYJ acknowledged that our additional ablations and novelty clarifications addressed most of their concerns. Following our further far-field forward-motion experiments, the reviewer found the additional results satisfactory and increased their rating from 2 to 3 (Weak Accept).

We sincerely thank all reviewers for their constructive feedback, which has substantially strengthened the experimental validation and presentation of our work. We also thank the AC for their time and efforts in coordinating the review process.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.