acceptodds
Withdrawn from ICLR 2027

MC-World: Memory-Guided Video Diffusion with Collision-Aware Exploration for Scene-Consistent 3D World Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion-based 3D scene generation has achieved remarkable progress. However, existing methods still face two critical challenges: (1) They struggle to remember the historical scenes and their spatial relationships in long-range video generation, making it difficult to maintain long-term 3D scene consistency; (2) They are prone to trajectory collisions with objects due to the lack of evasion capabilities for objects, leading to unreasonable geometric structures and inconsistent temporal sequences, thereby further limiting realistic and smooth scene exploration. To address these challenges, we propose MC-World, a novel memory-guided and collision-aware video diffusion framework for explorable 3D world generation. Specifically, MC-World consists of two core designs: Spatio-Temporal Memory Retrieval and Collision-Aware Exploration Control. The former retrieves temporally coherent and spatially relevant historical frames from a global memory cache, providing long-range contextual guidance for consistent multi-view generation. Collaboratively, the latter leverages collision detection strategy to adaptively truncate high-risk trajectory and switch generation directions, thus ensuring geometrically reasonable and temporally consistent scene content. Benefiting from the above designs, our method enables consistent 3D long-range scene generation and supports unrestricted realistic world exploration. Extensive experiments show that MC-World significantly superior to existing methods in various scenes.

Primary area:
generative models
Keywords:
Scene-Consistent, 3D World Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_36b5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for generating explorable 3D scenes from text. It first generates multi-view images from text, expands them into a panorama, and uses the it as the start for video generation. It then generates video frames along a given camera path and reconstructs a 3D scene from those frames.

To keep the generated video consistent, the paper proposes STMR, a more detailed method for retrieving past frames. To prevent the camera path from passing through objects, it introduces CAEC, which uses depth predictions from the video.

Critical strengths

The motivation is clear, and the method addresses it well. For example, STMR retrieves past frames more carefully to reduce the risk of forgetting earlier content. CAEC addresses problems such as passing through objects and shape distortions when generating videos for 3D reconstruction. Both modules show benefits in the qualitative and quantitative experiments.

Critical weaknesses
  1. The paper spends too much space describing existing methods, while giving too little detail about its own key modules and techniques. For example, lines 186–200 in the Preliminary section and the two related equations cover the basics of diffusion models. However, the technical details of this paper do not build on these equations. Similarly, lines 297–309 include two equations that describe the structure of an existing model. These details are not important for understanding the proposed method or framework. For the missing details that affect my understanding, please see “Questions For Authors.”

  2. Overall, the paper’s original contribution seems limited. Each part uses an existing model that is kept fixed, including text-to-image generation, inpainting, video generation, video depth estimation, and 3D reconstruction. The main contribution is therefore how these modules are connected to produce better results. Since the reasons for some design choices and their value remain unclear, I will consider the authors’ response before deciding whether the contribution is enough for a paper.

Questions for authors
  1. How do you keep the 4 images consistent with each other? Could the initial panorama contain visual inconsistencies?

  2. How are the 8 images in the memory bank related to the earlier panorama, and how are the camera poses for these 8 images determined or defined?

  3. The process for retrieving past frames seems quite complex. It starts with two candidate sets. The first includes the frame with the smallest camera angle difference and its neighboring frames. The second includes the top-k frames with the smallest combined distance based on camera angle and position. The method then takes the intersection of these two sets. This process is hard to follow, and it is not clear why each step is needed.

  4. The method uses a 2D Gaussian weight to assess collision risk. My concern is that, from a geometric perspective, if the camera path is not perpendicular to an object’s surface, the first collision may occur away from the image center. Distortions during generation may also appear earlier in those areas. Please explain the reasoning behind this design.

Reviewer_USmtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MC-World generates explorable 3D scenes by producing camera-conditioned videos and subsequently reconstructing them with 3D Gaussian Splatting. Starting from panoramic initialization, it retrieves historical frames using temporal neighborhoods and spatial pose proximity to condition video generation. A depth-based collision mechanism truncates high-risk trajectories, and the retained frames are used for memory updates and final 3D reconstruction. The paper reports improvements over existing scene-generation methods and ablations of its memory and collision modules.

Critical strengths

The paper addresses relevant practical challenges in video-based 3D scene generation: maintaining consistency across separately generated trajectories and avoiding degenerate observations caused by camera–object collisions. The reported results suggest that the implemented system produces visually promising scenes.

Critical weaknesses
  1. The methodological distinction from WorldExplorer [1] is insufficiently established. WorldExplorer already combines panoramic initialization, camera-guided video generation, retrieval of historical frames through scene memory, depth-based trajectory truncation, and final 3DGS reconstruction. MC-World’s identifiable differences primarily concern the retrieval rule and the use of trajectory-weighted median depth instead of central-crop average depth. However, the ablations remove memory or collision handling entirely rather than compare these specific changes against their WorldExplorer counterparts. They therefore do not establish the benefit of the claimed methodological advances.

[1] Schneider, Manuel-Andreas, Lukas Höllein, and Matthias Nießner. "Worldexplorer: Towards generating fully navigable 3d scenes." Proceedings of the SIGGRAPH Asia 2025 Conference Papers. 2025.

Questions for authors
  1. The methodological distinction from WorldExplorer [1] is insufficiently established. WorldExplorer already combines panoramic initialization, memory-conditioned video generation, depth-based trajectory truncation, and final 3DGS reconstruction. The apparent differences concern the retrieval rule and depth aggregation, but the ablations remove entire modules rather than compare these changes against their WorldExplorer counterparts. The benefit of the specific modifications therefore remains unclear.

  2. The reference images used to compute PSNR, SSIM, and LPIPS are not clearly specified. Without this information, it is difficult to determine whether these metrics measure fidelity to generated observations, cross-view consistency, or another property, and thus whether they support the claimed improvements in 3D consistency and geometric accuracy.

[1] Schneider, Manuel-Andreas, Lukas Höllein, and Matthias Nießner. “WorldExplorer: Towards Generating Fully Navigable 3D Scenes.” SIGGRAPH Asia, 2025.

Reviewer_anGgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MC-World, a memory-guided and collision-aware framework for video diffusion-based 3D world generation. It aims to address long-term scene inconsistency and trajectory-object collisions during scene exploration. Specifically, the method introduces Spatio-Temporal Memory Retrieval (STMR), which retrieves historical frames based on temporal continuity and spatial pose proximity, and Collision-Aware Exploration Control (CAEC), which estimates collision risks using trajectory-aligned depth statistics and truncates potentially risky trajectories. The preserved frames are then used for subsequent memory retrieval and 3D Gaussian Splatting reconstruction. The method is evaluated on several 3D scene generation benchmarks and compared with existing approaches.

Critical strengths

The paper considers two important challenges in video diffusion-based 3D world generation. Maintaining long-term scene consistency and avoiding trajectory-object collisions are both relevant to generating explorable 3D environments.

The proposed STMR and CAEC have intuitive motivations. Combining temporal and spatial information for historical frame retrieval is reasonable, while using trajectory-aware depth statistics to identify potential collisions provides a lightweight way to regulate scene exploration.

The experimental results show consistent improvements over existing methods across several reported metrics. The ablation study also suggests that both STMR and CAEC contribute to the overall performance.

Critical weaknesses

The overall framework is highly similar to WorldExplorer, which already incorporates historical scene memory retrieval and depth-based collision detection into a video diffusion-based 3D scene generation pipeline. In particular, both methods maintain a global memory of historical frames and camera poses, retrieve relevant historical views for subsequent generation, and use collision detection to truncate invalid trajectories. Therefore, the main contributions of this paper seem to be incremental extensions of an existing framework, rather than a substantially new approach to memory-guided 3D world generation.

The proposed improvements in STMR and CAEC are relatively conventional. STMR combines temporal neighborhood retrieval with spatial pose-based top-K selection, while CAEC replaces the center-region depth average with trajectory-aligned Gaussian weighting and a weighted median. Although these modifications are intuitively reasonable, the paper does not sufficiently explain why these particular combinations can effectively address the limitations of existing methods. For example, it is unclear why intersecting the temporal and spatial candidate sets is more effective than directly retrieving spatially nearest historical views. Similarly, the advantages of the proposed weighted median over existing depth-based collision detection are not sufficiently motivated. Overall, the methodological novelty appears limited.

Although the paper compares the complete framework with WorldExplorer, the experimental results do not sufficiently isolate the improvements brought by the proposed modules over the corresponding components in WorldExplorer. The ablation study in Table 2 only evaluates the removal of STMR and CAEC, which demonstrates the usefulness of these modules in the proposed framework, but does not establish that the proposed designs are better than the existing alternatives. A more direct comparison by replacing the corresponding WorldExplorer components with STMR and CAEC would be important to verify whether the proposed modifications actually improve historical retrieval and collision handling.

The evaluation of 3D consistency is not sufficiently clear. The paper reports PSNR, SSIM, and LPIPS on renderings from multiple novel viewpoints, but does not clearly specify the reference images used to calculate these metrics or how the evaluation views are constructed.

Questions for authors

The paper makes strong claims about supporting unrestricted exploration along arbitrary camera trajectories. However, the implementation details in the supplementary material indicate that the experiments mainly use four predefined trajectories (in, left, right, and up). Moreover, CAEC primarily detects potential collisions and truncates the remaining trajectory, rather than explicitly planning an alternative collision-free path. Therefore, the current experimental results do not fully support the claims regarding unrestricted exploration and adaptive trajectory control.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.