- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a method for generating explorable 3D scenes from text. It first generates multi-view images from text, expands them into a panorama, and uses the it as the start for video generation. It then generates video frames along a given camera path and reconstructs a 3D scene from those frames.
To keep the generated video consistent, the paper proposes STMR, a more detailed method for retrieving past frames. To prevent the camera path from passing through objects, it introduces CAEC, which uses depth predictions from the video.
Critical strengths
The motivation is clear, and the method addresses it well. For example, STMR retrieves past frames more carefully to reduce the risk of forgetting earlier content. CAEC addresses problems such as passing through objects and shape distortions when generating videos for 3D reconstruction. Both modules show benefits in the qualitative and quantitative experiments.
Critical weaknesses
-
The paper spends too much space describing existing methods, while giving too little detail about its own key modules and techniques. For example, lines 186–200 in the Preliminary section and the two related equations cover the basics of diffusion models. However, the technical details of this paper do not build on these equations. Similarly, lines 297–309 include two equations that describe the structure of an existing model. These details are not important for understanding the proposed method or framework. For the missing details that affect my understanding, please see “Questions For Authors.”
-
Overall, the paper’s original contribution seems limited. Each part uses an existing model that is kept fixed, including text-to-image generation, inpainting, video generation, video depth estimation, and 3D reconstruction. The main contribution is therefore how these modules are connected to produce better results. Since the reasons for some design choices and their value remain unclear, I will consider the authors’ response before deciding whether the contribution is enough for a paper.
Questions for authors
-
How do you keep the 4 images consistent with each other? Could the initial panorama contain visual inconsistencies?
-
How are the 8 images in the memory bank related to the earlier panorama, and how are the camera poses for these 8 images determined or defined?
-
The process for retrieving past frames seems quite complex. It starts with two candidate sets. The first includes the frame with the smallest camera angle difference and its neighboring frames. The second includes the top-k frames with the smallest combined distance based on camera angle and position. The method then takes the intersection of these two sets. This process is hard to follow, and it is not clear why each step is needed.
-
The method uses a 2D Gaussian weight to assess collision risk. My concern is that, from a geometric perspective, if the camera path is not perpendicular to an object’s surface, the first collision may occur away from the image center. Distortions during generation may also appear earlier in those areas. Please explain the reasoning behind this design.