- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SEG for successive novel view synthesis with incremental and unordered viewpoint queries. Its key idea is a self-expanding geometric state (S_k=(Z_k,M_k)), which combines geometric latents extracted from DA3 with KV reference memory to reuse previously generated content. The framework further introduces causal-prefix supervision to avoid future-view information, causal geometric diffusion for state expansion, and a Geometric Adapter for RGB readout.
Experiments on RE10K and DL3DV show improvements in synthesis quality, camera pose accuracy, and inference efficiency over the evaluated baselines. Ablations further evaluate the contributions of the state design, causal-prefix supervision, and robustness to unordered queries.
Critical strengths
- Meaningful problem setting and reasonable geometric state design.
The paper addresses successive NVS with incremental and unordered viewpoint queries, which is practically relevant for interactive scene exploration. Leveraging geometry-pretrained latents with causal-prefix supervision and reusable KV memory provides a reasonable approach to accumulating generated context while maintaining causal conditioning.
- Promising empirical results.
SEG consistently improves image quality and pose accuracy over the evaluated baselines on RE10K and DL3DV, while achieving lower inference cost than the compared generative approaches. The component ablations and unordered-query evaluation further support the effectiveness of the proposed pipeline.
Critical weaknesses
- Insufficient comparison with closely related autoregressive NVS methods.
CausNVS (Kong et al., 2025) already supports incremental and unordered viewpoint queries through causal diffusion and pose-aware KV caching. Although cited, it is not included in the experimental comparisons. This makes it difficult to assess SEG's improvement over existing methods addressing essentially the same problem setting.
- Limited evidence for the contribution of geometric latent representation.
The main distinction of SEG from existing causal NVS methods (e.g. CausNVS) lies in generating and retaining geometry-pretrained latents rather than conventional image latents. However, the current ablations mainly validate state expansion and individual components, without isolating the advantage of geometric latents over conventional latent representations with comparable causal KV memory. Thus, the effectiveness of the paper's most distinctive technical contribution remains insufficiently established.
Questions for authors
See weakness.
5 comments
Official Response to Reviewer oM3i (Part 2/2)
W2. About geometric representation
Thanks. During the rebuttal period, we have conducted experiments comparing a 2D VAE variant with SEG. Specifically, we replace the SEG state with the 2D VAE latents of Stable Diffusion XL [1] while keeping all other settings unchanged. The results are shown in the following table:
| Dataset | Settings | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| RE10K | w/ 2D VAE state | 13.82 | 0.528 | 0.597 |
| RE10K | w/ SEG state (ours) | 14.68 | 0.542 | 0.459 |
| DL3DV | w/ 2D VAE state | 13.74 | 0.444 | 0.639 |
| DL3DV | w/ SEG state (ours) | 15.08 | 0.461 | 0.472 |
Despite using the same causal pipeline and training strategy, the 2D VAE variant produces lower-quality renderings. We attribute this gap to poorer alignment with the queried camera poses during successive and unordered queries. In addition to rendering quality, we evaluate geometric consistency through camera pose accuracy, which is particularly important for constructing spatially coherent scene representations. We follow the evaluation protocol used in Table 4 of our paper and report the results below:
| Dataset | Settings | ATE ↓ | RPE-t ↓ | RPE-r ↓ |
|---|---|---|---|---|
| RE10K | w/ 2D VAE state | 0.378 | 0.506 | 3.129 |
| RE10K | w/ SEG state (ours) | 0.157 | 0.204 | 1.687 |
| DL3DV | w/ 2D VAE state | 0.499 | 0.582 | 4.420 |
| DL3DV | w/ SEG state (ours) | 0.133 | 0.185 | 1.322 |
The 2D VAE variant also performs substantially worse in camera pose accuracy. This indicates that the choice of state representation contributes to viewpoint accuracy beyond the benefits of history reuse and causal modeling. In particular, geometric features provide spatial cues that help align successive predictions with the queried camera poses. These results are now included in Section 4.4 of the revised paper, i.e. Table 9 reports 2D image quality, and Table 10 reports camera pose accuracy.
We have also added Figure 6 in Section 4.4 to compare the rendered views and estimated camera poses of the two state representations. Our SEG preserves the arrangement of scene structures and their alignment with the queried viewpoints more consistently across successive generations.
[1] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. International Conference on Learning Representations (ICLR), 2024.
Dear Reviewer oM3i,
Thank you for your thoughtful review and constructive suggestions, which have significantly improved the quality and clarity of our paper. We sincerely hope that our rebuttal has fully addressed all of your concerns. As the discussion period is ending soon, if you need any further clarification or additional information, please do not hesitate to contact us. We truly appreciate your time and effort devoted to reviewing our work.
Best regards, Authors of Submission 21404
Official Response to Reviewer oM3i (Part 1/2)
We sincerely thank Reviewer oM3i for the constructive review and insightful questions. We address all the points raised in review as follows:
W1. Comparison with CausNVS
Thank you for raising this comparison. Besides the choice of geometric state representation, we highlight two further differences:
Original motivation. Our key insight is that retained history should support successive queries across geometrically unordered viewpoints. This motivates our joint design of a geometric state and causal-prefix supervision to maintain geometric consistency while supporting incremental queries. CausNVS focuses on streaming queries, whereas our work emphasizes geometric reasoning across successively queried, unordered viewpoints, with carefully designed evaluation settings, 3D metrics and comparisons.
Causal exploration with a geometric state. While CausNVS uses a causal pipeline with 2D VAE latents, our geometric state introduces additional challenges: maintaining representation stability during state expansion and preventing future-view information leakage during training. We address both through our geometric state design and causal-prefix supervision, and evaluate their effectiveness experimentally.
We have expanded the CausNVS discussion in Section 2 (Related Work) and Appendix E (Discussion).
As we could not access an official implementation or pretrained checkpoints for CausNVS through its project page or public repositories when conducting these experiments, we reimplemented CausNVS based on the original paper. We compare this reimplementation with SEG under the evaluation protocol of Table 3 in our paper:
As shown above, SEG achieves better performance than our CausNVS reimplementation on both datasets. We further assess 3D geometric consistency through camera pose accuracy as reported below, which verifies our claim that geometry is essential in successive NVS:
We have incorporated these comparisons into Table 3 for image quality and Table 4 for camera pose accuracy.