acceptodds
Withdrawn from ICLR 2027

GSExplorer: Interactive World Generation with a Persistent 3D Gaussian World State

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive world generation requires both accurate local control and stable scene consistency over long rollouts. Existing methods improve local generation through camera or geometry conditioning and extend historical context with memory mechanisms. However, long-horizon exploration remains challenging as target-view guidance becomes incomplete and inconsistencies accumulate across repeated scene updates. We present GSExplorer, a generation–reconstruction framework built around a persistent 3D Gaussian world state. At each step, the current 3DGS scene provides multimodal rendering conditions for a condition-driven video generator, while camera geometry is injected through a separate Relative Ray Encoding branch. Generated videos are then fed back to a SLAM-based reconstruction backend, where inconsistency filtering prevents conflicting content from corrupting the scene and cross-chunk bundle adjustment improves spatial alignment across generation steps. We further design a data-construction pipeline that reproduces realistic 3DGS degradation during scene extrapolation, yielding 293K training pairs of degraded multimodal renderings and real target videos. GSExplorer achieves the best overall performance among the compared methods on both single-step and three-step WorldScore evaluation, with strong camera control and consistency across both settings, including a Camera Control score of 99.07 in single-step extrapolation. On 481-frame leave-and-return rollouts, it also shows strong revisit consistency, demonstrating the value of maintaining a reliable 3D world state for continued exploration.

Primary area:
generative models
Keywords:
Interactive World Model, Video Generation, 3DGS scene generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GprSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GSExplorer is a camera-controlled video world model built around a persistent 3D Gaussian scene. Multimodal renderings from the scene and a separate camera-encoding branch condition a Wan-based video generator. Generated frames are then integrated into the scene through a SLAM backend, with inconsistency filtering to protect existing content and cross-chunk bundle adjustment to improve alignment. The paper also constructs 293K training pairs of degraded renderings and real videos. Experiments evaluate single-step and multi-step generation on WorldScore and revisit consistency over 481-frame rollouts.

Critical strengths

1. The system addresses a practical problem in maintaining scene memory during generation. Generated observations can disagree with earlier views, so directly incorporating them into the scene can corrupt the conditions used for subsequent generation. Filtering conflicting observations while allowing new regions to be added is a reasonable way to protect the accumulated scene. Combining this with camera refinement gives the system a clear strategy for maintaining its world state across chunks.

2. The experiments show good camera control and revisit consistency. GSExplorer achieves the highest average scores among the compared methods in both WorldScore settings and reaches 19.85 dB PSNR on the 481-frame revisit evaluation. Table 5 also shows that filtering and cross-chunk BA both contribute to the results, supporting the usefulness of these components within the system.

Critical weaknesses

1. It is unclear what the world-update method adds beyond integrating an existing SLAM backend. Appendix A.1.1 states that the system builds on ARTDECO, which already supports incremental Gaussian mapping, historical keyframe association, loop closure, and global pose optimization. Section 3.1 does not clearly explain what changes to these mechanisms are needed for generated videos. The inconsistency filter is a straightforward appearance-thresholding rule, while the cross-chunk BA objective is described only in general terms. Removing BA in Table 5 shows that pose refinement is useful, but does not establish an advance over the existing backend. I would like to see the same generator connected to the original ARTDECO backend, followed by the proposed modifications, to make the technical contribution and its gains clear.

2. The contribution relative to prior work on reconstructing inconsistent generated views needs clarification. World Reconstruction From Inconsistent Views (ECCV 2026) directly addresses this problem through non-rigid alignment and deformation-aware reconstruction, but is not discussed here. This distinction matters because generated objects can change shape across views, which camera-pose refinement alone cannot generally correct. GSExplorer instead rejects observations that conflict with the existing map, leaving unclear how much useful information is lost and which types of inconsistency it can handle. Although the ECCV work focuses on reconstruction rather than the full generation loop, a comparison on the same generated sequences would help establish what GSExplorer’s update strategy offers beyond existing treatments of inconsistent views.

3. Runtime and total system costs are missing. Each generation chunk requires camera and depth estimation, correspondence matching, pose refinement, and Gaussian scene optimization, in addition to video generation. Appendix A.3 acknowledges this overhead, but provides no end-to-end timing or cost breakdown. The system also retains the Gaussian map, historical mapping views, camera graph, and retrieval database, without reporting how their storage and update costs change over long rollouts. For a system presented as interactive world generation, these measurements are essential to understanding whether the quality gains come at a practical computational cost.

Questions for authors
  1. What changes were made to ARTDECO beyond retaining its state across chunks? With the same generator, how does the original backend compare with the proposed update strategy?

  2. How does the method handle local geometric deformation that cannot be corrected by changing camera poses? Could you compare against World Reconstruction From Inconsistent Views on the same generated observations, including reconstruction quality and computational cost?

  3. What is the end-to-end latency per chunk, and how much time is spent on generation versus scene updating? Please specify the hardware, resolution, and sampling settings, and report total memory usage and update time as the rollout grows.

Reviewer_UGnPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

GSExplorer enables camera- and text-controlled world generation from a single image through an iterative generation–reconstruction loop. A persistent 3D Gaussian scene supplies RGB, depth, opacity, and hole-mask conditions to a Wan2.2-based video generator, while a separate branch provides relative-ray camera conditioning. Generated observations are incorporated into the scene through inconsistency filtering and cross-chunk bundle adjustment. The authors also construct approximately 293K training pairs of degraded renderings and target videos. Evaluation comprises single-step and three-step WorldScore experiments, 481-frame leave-and-return trajectories across 50 cases, and component ablations.

Critical strengths

The ablations show that filtering and cross-chunk alignment benefit the proposed implementation. Removing filtering lowers the three-step WorldScore average from 71.56 to 65.14, while removing bundle adjustment lowers it to 68.91; revisit performance also deteriorates. However, these results primarily validate the system’s internal design choices. They do not, by themselves, demonstrate a substantive methodological advance or an advantage over existing approaches. I therefore consider them useful implementation-level validation, but not a compelling strength supporting acceptance.

Critical weaknesses
  1. Both central elements of GSExplorer have substantial precedent. Bootstrap-GS constructs degraded-rendering/ground-truth pairs, fine-tunes a diffusion model to regenerate novel-view renderings, and incorporates the regenerated views into Gaussian optimization [1]. Difix3D+ explicitly alternates between diffusion-based artifact removal and distillation of corrected views into the 3D representation [2]. ArtiFixer extends this approach with opacity-aware conditioning, camera and reference-view guidance, autoregressive video generation, and generated-view supervision to refine the underlying representation [3]. Similarly, 3DGS-Enhancer and GSFixer use video diffusion to restore corrupted Gaussian renderings and improve reconstruction [4,5]. Learning to repair degraded 3DGS renderings, including through paired degraded/clean training data, is therefore an established strategy.

Iterative scene expansion through alternating generation and reconstruction also has direct precedents. GenFusion trains a video diffusion model conditioned on artifact-prone RGB-D renderings and incorporates restored frames through a cyclical reconstruction–generation pipeline [6]. Scene Splatter maintains global Gaussian representations, refines them with enhanced video frames, and renders conditions for subsequent generation iterations [7]. Gen3C, ViewCrafter, and HunyuanWorld-Voyager likewise condition generation on explicit geometry or world caches to progressively extend the represented scene [8–10].

Against this background, the manuscript does not clearly establish what constitutes a substantive methodological advance. Combining multimodal conditioning, RRE, filtering, and cross-chunk bundle adjustment may be practically useful, but adopting Gaussian primitives does not itself introduce a new spatial-memory mechanism. A stronger contribution would require a clearly differentiated conditioning or scene-update mechanism, supported by controlled comparisons. The approximately 293K training pairs represent substantial engineering effort, but the underlying construction principle is also well precedented. In particular, the contribution of the Gaussian-birth-based degradation procedure remains unclear without analysis or ablation against existing degradation strategies.

  1. The evaluation should directly compare the proposed mechanisms with relevant prior approaches: rendering-restoration methods for the repair component and persistent-geometry generation pipelines for end-to-end exploration. Although Voyager, Spatia, and WonderWorld are included, closely related methods such as GenFusion, Scene Splatter, and Gen3C are absent. Comparisons on shared tasks, with differences in backbone, training data, and reconstruction procedures clearly documented, would help isolate the proposed method’s contribution. Ablating filtering or bundle adjustment establishes their utility within GSExplorer, but does not demonstrate an advantage over a strong alternative scene-update pipeline.

The aggregate WorldScore results also do not establish consistently superior geometric consistency. GSExplorer achieves the highest reported single-step 3D Consistency score, but its three-step score is 77.50, below Uni3C’s 79.95. Moreover, the 481-frame evaluation measures similarity between initial and final images across 50 cases and compares against only four baselines. This provides evidence of endpoint appearance preservation, but leaves consistency throughout exploration—and across repeated observations of newly generated regions—insufficiently assessed. Given the limited methodological differentiation, the current empirical evidence does not provide a sufficiently strong basis for acceptance.

[1] Bootstrap-GS: Self-Supervised Augmentation for High-Fidelity Gaussian Splatting. arXiv:2404.18669, 2024.

[2] Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models. CVPR 2025. arXiv:2503.01774.

[3] ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Models. SIGGRAPH 2026. arXiv:2603.00492.

[4] 3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors. NeurIPS 2024. arXiv:2410.16266.

[5] GSFixer: Improving 3D Gaussian Splatting with Reference-Guided Video Diffusion Priors. arXiv:2508.09667, 2025.

[6] GenFusion: Closing the Loop between Reconstruction and Generation via Videos. CVPR 2025. arXiv:2503.21219.

[7] Scene Splatter: Momentum 3D Scene Generation from Single Image with Video Diffusion Model. CVPR 2025. arXiv:2504.02764.

[8] GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control. CVPR 2025. arXiv:2503.03751.

[9] ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis. arXiv:2409.02048, 2024.

[10] Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation. arXiv:2506.04225, 2025.

Questions for authors
  1. What specific mechanism distinguishes GSExplorer from the diffusion-based rendering restoration and iterative reconstruction–generation pipelines demonstrated by Bootstrap-GS, Difix3D+, GenFusion, and Scene Splatter? Please clarify the methodological contribution beyond the choice of representation and the integration of existing components.

  2. What controlled experiments isolate the benefits of the Gaussian-birth-based degradation procedure and the proposed world-update mechanism? Can the authors compare these components with existing degradation/restoration strategies and closely related scene-expansion pipelines under shared task settings?

  3. Beyond similarity between the initial and final images, what evidence supports geometric consistency throughout the rollout, particularly when revisiting regions generated after initialization?

Reviewer_a7HhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GSExplorer, a generation-reconstruction framework for interactive world generation built around a persistent 3D Gaussian Splatting (3DGS) world state. To prevent error accumulation over long rollouts, the framework decouples spatial memory from local video synthesis. At each step, a condition-driven video generator creates novel views based on text prompts, multimodal 3DGS renderings, and a separate camera geometry branch. The generated videos are then selectively integrated back into the shared 3DGS scene through a SLAM-based backend that applies inconsistency filtering and cross-chunk bundle adjustment for precise alignment. The primary contributions are three-fold: (1) a tightly coupled video generation and 3DGS-SLAM reconstruction loop; (2) a data-construction pipeline that simulates realistic 3DGS degradation to yield a 293K training dataset; and (3) state-of-the-art performance on the WorldScore and R2M-Bench benchmarks. The method is validated through comprehensive quantitative and qualitative comparisons against recent interactive world models, alongside ablation studies verifying the impact of each conditioning modality and backend component.

Critical strengths

1. Clear and well-structured presentation. The paper is well-written and easy to follow. The introduction effectively points out the existing research gaps and presents a clear, intuitive research question. The overall framework and its methodology are presented in a highly accessible manner, making the authors' core ideas immediately understandable to the reader.

2. Meticulously designed integration pipeline. The paper introduces a sophisticated and comprehensive system framework for interactive world generation. By seamlessly coupling a condition-driven video generator—which balances both camera geometry and multi-modal rendering inputs—with a SLAM-based backend that handles cross-chunk and revisit constraints, the proposed method provides a well-engineered solution that addresses both camera controllability and long-horizon scene persistency.

3. Effective large-scale data pipeline. The authors propose a dedicated data construction pipeline that reproduces realistic 3DGS degradation while maintaining geometric alignment with real target videos. By leveraging this pipeline to curate a large-scale training dataset of 293K video pairs, the paper provides a solid data-driven foundation that enables the model to learn robust scene completion and control.

Critical weaknesses

1. Unclear novelty of individual components. While the framework is well-engineered from a system-integration perspective, the conceptual novelty of each individual module seems to be incremental and requires further clarification against existing works. In the world-state update component, updating an explicit 3D spatial memory using a support mask is already a common and established practice in the literature. Furthermore, the usage of boundary frames and revisit loop matches to perform optimization borrows standard techniques from classical SLAM. The authors need to clarify what uniquely distinguishes their specific tracking and filtering formulations from these closely related state-of-the-art baselines. (Moreover, regarding the methodology and novelty of the video generation component, please refer to Question 1 for detailed inquiries.)

2. Underperformance of several individual metrics in Table 2. In Table 2 (three-step extrapolation), although the proposed method achieves the best overall average score, it underperforms in several key individual metrics that are tightly coupled with the paper's core claims. Specifically, 3D Consistency and Camera Control—the two most critical pillars for verifying stable world state maintenance and precise trajectory following—rank only second. Notably, the gap in the Camera Control score compared to the best-performing baseline is substantial. This underperformance in multi-step generation contradicts the narrative that the proposed persistent 3DGS world state inherently prevents trajectory drift and error accumulation better than all existing alternatives.

3. Amount and justification for runtime overhead are needed. The authors acknowledge in the limitations section that the introduction of camera estimation and 3DGS scene refinement adds computational runtime compared to purely feed-forward video generation. However, the paper does not provide any quantitative data regarding this overhead, such as a percentage increase in wall-clock time relative to one of the compared video generation baselines. Without a thorough trade-off analysis between the achieved performance gains and the sacrificed efficiency, it is difficult to evaluate whether the heavy backend optimization loop is practical or justifiable for interactive world generation tasks.

Questions for authors

Q1. To substantiate the claim that explicitly conditioning the model on both 3DGS rendering cues and camera parameters provides architectural distinction, could the authors provide a deeper rationale for their specific design choices? For example, what was the intent behind feeding the multimodal rendering features into the front and residual sites of the DiT blocks while adding the RRE attention outputs in parallel within the self-attention layer? Otherwise, the authors can provide a quantitative comparison against reasonable architectural alternatives, as the current ablation study only tests the complete removal of these components.

Q2. Regarding the construction of the support mask detailed in Section 3.1, what was the rationale or empirical intent behind selecting the specific values for the historical coverage threshold (η\eta), the discrepancy scaling factor (λ\lambda), and the consistency threshold (τc\tau _{c})? Providing a sensitivity analysis showing how the final performance would be affected would greatly enhance the completeness and soundness of the methodology.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.