acceptodds
Under review as a conference paper at ICLR 2027

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

Abstract

Novel view synthesis from sparse images must reconcile faithful reconstruction of observed regions with plausible completion of unseen content, while maintaining world consistency across viewpoints. Existing geometry-based methods preserve observed scene structure but often struggle to complete unseen regions, whereas video generative models offer rich appearance priors but accumulate inconsistencies during sequential view generation. We propose GeoVerse, a framework that synthesizes world-consistent novel views by performing generation within the geometric latent space of a pretrained 3D foundation model and injecting appearance priors from a video generative model. Specifically, GeoVerse extracts multilevel features from Wan2.2 VACE and injects them into the geometric latent diffusion model via a ControlNet-style adapter, incorporating video-learned appearance priors to enhance structural completion. To enforce cross-view coherence, a global spatial memory continuously aggregates observed and synthesized content, reprojecting target-aligned guidance to anchor subsequent predictions to a shared scene representation. Extensive experiments across diverse datasets demonstrate improved visual quality and geometric consistency, with a 2.23 dB higher PSNR on DL3DV and 32.4% lower ATE on Mip-NeRF360 compared to GLD.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel View Synthesis, Geometric Latent Space, Spatial Memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_u7eSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper’s goal is world-consistent novel view synthesis from sparse images, which requires both faithfully reconstructing observed regions and plausibly completing unseen ones. Geometry-based methods preserve structure but cannot generate missing content, while video diffusion models offer rich appearance priors but drift across views. GeoVerse addresses this by performing diffusion in the geometric latent space of a frozen 3D foundation model (DA3). It injects appearance priors from a frozen video model (Wan2.2 VACE) through a ControlNet-style adapter. It also maintains a persistent point-cloud spatial memory, re-projected into each target view as RGB-D guidance, to keep multi-round generation consistent. The work shows that video priors can enhance geometric latent generation without adopting a sequential video pipeline, offering an efficient path toward world-consistent scene synthesis.

Critical strengths

Consistent gains across in-domain and out-of-domain benchmarks. GeoVerse outperforms a broad set of generative NVS baselines on RealEstate10K, DL3DV, Mip-NeRF 360 and ScanNetv2, with clear improvements over GLD in both visual quality and pose accuracy (e.g., +2.45 dB PSNR and 32.4% lower ATE on Mip-NeRF 360). This is consequential to my assessment because the gains hold on out-of-domain data, which suggests that enriching the geometric latent space with video priors improves generalization rather than fitting the training distribution.

Spatial memory that extends single-pass consistency to multiple rounds. The persistent point-cloud memory re-projects accumulated content into each target view as RGB-D guidance with a validity mask, anchoring new predictions to a shared scene representation. This matters because drift in sequential view generation is a key weakness of prior video-based approaches. The ablation shows that removing the memory clearly degrades both fidelity and cross-view alignment, and the long-sequence results on ScanNetv2 (15.33 → 19.65 dB PSNR over GLD) support its effectiveness.

Critical weaknesses

1. Limited novelty, and the contribution of the video prior is not fully isolated. Most of the pipeline is built from existing components: the GLD backbone, a point-cloud memory similar to ViewCrafter, a ControlNet-style adapter, and piecewise reflow distillation. The main new idea is injecting single-pass Wan2.2 features into the geometric latent space. The ablation in Table 3 suggests that this prior is the most important component, but it is reported only on ScanNetv2, and the paper does not state whether the ablated variants were trained on the same 15 datasets. Since the training data was also expanded from the 4 datasets used by GLD, I wonder how much of the overall gain over GLD comes from the video prior and how much from the larger training set. This matters because the video prior is the core contribution, and a clearer isolation of its effect would make the claim much more convincing.

2. Mixed evidence for world consistency. GeoVerse achieves the best ATE and MEt3R across benchmarks, which is a clear strength. However, on reprojection error it is behind some baselines on several datasets, and the gains on cross-view consistency metrics are generally smaller than those on image quality. Given that world consistency is the headline claim, I wonder whether the improvements mainly reflect better per-view image quality rather than better 3D consistency. A short discussion of why the reprojection error does not improve would help clarify this.

Questions for authors

Ablation settings. Were the variants in Table 3, especially w/o Wan2.2 prior, trained on the same 15 datasets and with the same distillation as the full model? If possible, could you also report the main ablations on one more benchmark, such as DL3DV or Mip-NeRF 360? Does the Wan prior help in unobserved regions? Could you report metrics separately on the valid (projected) and invalid (unprojected) target regions, with and without the Wan prior? This would directly support the claim that video features help complete unseen content. Choice of extraction timestep. How was t = 500 chosen for Wan feature extraction? A brief explanation, or a small sweep over a few timesteps, would help show how sensitive the results are to this choice. Source of the speedup. The speedup over GLD seems to come mainly from reflow distillation and the one-step L0 cascade, which could also be applied to GLD. Could you provide a per-component runtime breakdown and the peak GPU memory, and comment on how GLD would perform with the same few-step sampling?

2 comments

We sincerely thank Reviewer u7eS for the careful assessment of novelty, fair video-prior attribution, geometric consistency, cross-dataset ablations, and computational cost. The comments encouraged us to provide clearer component attribution, analyze reprojection-error limitations, and add detailed runtime, memory, and few-step diagnostics. Below are our point-by-point responses:

W1 / Q1 :

GeoVerse integrates large-scale geometric-latent modeling, frozen video-prior transfer, persistent spatial memory, and few-step distillation to jointly synthesize RGB and geometry rather than merely refining an explicit 3D reconstruction.

Across ablation variants trained on the same 15-dataset pool, scaling training data improves both in-domain and out-of-domain performance, while incorporating the Wan prior yields an even larger gain. This improvement is especially pronounced on the out-of-domain ScanNetv2 dataset, where multi-round generation progressively exposes unobserved regions and heightens reliance on a strong generative prior. Table 6 further disentangles the contributions of the video prior, target hints, injection design, and training losses on DL3DV.

Table 6. Ablation comparison on the DL3DV dataset.

VariantPSNR ↑SSIM ↑LPIPS ↓ATE ↓RPEr ↓RPEt ↓Reproj. ↓MEt3R ↓
GeoVerse19.610.5980.3390.0280.9420.0600.6490.261
w/o Wan2.2 prior18.220.5550.3670.0451.2040.1090.6530.263
w/o target hint18.590.5630.3710.0401.1150.0950.6510.260
MoT video-KV18.190.5520.3800.0471.2290.1160.6560.268
w/o RGB loss19.150.5890.3540.0321.0190.0610.6500.260
with depth loss19.190.5950.3500.0331.0390.0690.6500.262

W2 :

We agree that reprojection error does not uniformly favor GeoVerse. Its evaluation estimates geometry and cameras from the generated images, so the residual also depends on appearance and correspondence support. Smaller realized viewpoint changes or smooth, low-texture outputs can yield lower residuals without more faithful target-view synthesis. Appendix Fig. 10 illustrates these visual confounds.

Q2 :

Among reference-valid target pixels, initially observed reconstruction regions and unobserved generation regions account for 63.19% / 36.81% on DL3DV, 75.40% / 24.60% on RE10K, and 62.32% / 37.68% on Mip-NeRF360, respectively. To reflect both components, we evaluate reconstruction fidelity with PSNR/SSIM and generation quality with FID/KID, as shown in Table 7. Comparing variants with and without the Wan prior further indicates that video features improve both novel-content synthesis and reconstruction fidelity through appearance and viewpoint cues. This region-wise evaluation more clearly separates reconstruction from generation capability, and GeoVerse achieves the strongest performance in both aspects across all three datasets.

Table 7. Quantitative comparison of fidelity in reconstructed regions and quality in generated regions. Recon. reports PSNR ↑ / SSIM ↑; Gen. reports FID ↓ / KID ↓.

MethodDL3DV Recon.DL3DV Gen.RE10K Recon.RE10K Gen.Mip-NeRF360 Recon.Mip-NeRF360 Gen.
GLD19.51 / 0.610197.8 / 0.03522.67 / 0.766199.7 / 0.08819.05 / 0.433246.3 / 0.058
ViewCrafter17.15 / 0.565237.4 / 0.07317.74 / 0.672282.2 / 0.21516.81 / 0.403344.8 / 0.137
NeoVerse12.55 / 0.365210.7 / 0.04612.61 / 0.477224.9 / 0.12613.77 / 0.302271.6 / 0.076
GEN3C18.57 / 0.622263.1 / 0.09920.91 / 0.756240.6 / 0.16818.29 / 0.452326.7 / 0.117
MVGenMaster19.48 / 0.649185.9 / 0.02522.84 / 0.780261.8 / 0.16218.27 / 0.433249.2 / 0.055
Matrix3D14.42 / 0.446243.8 / 0.04617.61 / 0.605385.0 / 0.43016.13 / 0.341259.7 / 0.092
CAMEO11.26 / 0.400255.8 / 0.07813.91 / 0.523217.7 / 0.09811.44 / 0.261395.1 / 0.201
Ours w/o Wan2.2 prior20.04 / 0.643178.5 / 0.03223.19 / 0.791175.1 / 0.07419.39 / 0.441232.0 / 0.054
Ours21.06 / 0.654142.3 / 0.02024.27 / 0.806148.0 / 0.05620.53 / 0.481201.2 / 0.045

Q3 :

We adopt t=500 as a fixed intermediate timestep for extracting video-prior features. Because the input combines reliable projected content with unobserved regions requiring synthesis, an intermediate conditioning level naturally balances content preservation and generative completion. Preliminary experiments indicated low sensitivity around this value, so we kept this setting fixed across all subsequent runs.

Q4.1 :

Table 8 provides a per-component runtime breakdown, recording a peak GPU memory of 23.03 GiB on a single NVIDIA A800 80GB GPU using two context images and six target views at 504×504 resolution.

Table 8. Detailed breakdown of CUDA-synchronized model-inference latency.

Model inference stageTime (s)
Context DA3 geometry0.32
Multi-level feature encoding0.55
Wan feature extraction0.25
L1 generation5.94
Cascade + decode2.17
Total model inference9.23

Q4.2 :

To quantify the effect of distillation, we directly applied 3+1 updates to pretrained GLD weights without trajectory distillation. Table 9 shows that full-image quality degrades across all four datasets, while Appendix Fig. 11 further exposes clear losses in newly synthesized detail. Therefore, merely shortening the sampling schedule fails to reproduce the distilled model's joint image and geometry behavior. These results confirm the necessity of explicitly training the geometric generator for the target few-step trajectory.

Table 9. Quantitative comparison of GLD performance using the sampling configuration reported in its paper (50/50, 49+49=98 denoising steps) and few-step sampling (4/2, 3+1=4 denoising steps).

DatasetPointsPSNR ↑SSIM ↑LPIPS ↓ATE ↓RPEr (°) ↓RPEt ↓Reproj. ↓MEt3R ↓
DL3DV50/5017.380.5460.3830.0581.5380.1250.6520.262
DL3DV4/217.880.5710.3680.1123.7170.1640.6370.263
RE10K50/5019.660.7090.2990.0371.3380.0670.6080.183
RE10K4/220.130.7380.2830.0401.2590.0740.6010.178
Mip-NeRF36050/5017.570.4400.4060.1052.3370.2020.6210.251
Mip-NeRF3604/218.100.4620.3940.34512.2760.7270.6020.261
ScanNetv250/5015.330.6640.4910.0110.3960.0190.5980.108
ScanNetv24/215.840.6810.4700.0250.9160.0380.6000.124
Reviewer_NxPVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines geometric foundation models and video diffusion priors to provide strong conditioning for GLD, improving novel view synthesis quality and 3D consistency. The overall pipeline integrates several existing techniques and achieves promising performance.

Critical strengths
  1. The authors introduce video diffusion priors into GLD and achieve clear improvements in reconstruction quality.

  2. The engineering effort is substantial, and scaling up the training data is also a valuable contribution to this field.

Critical weaknesses
  1. The overall idea of this paper has already been explored in previous works. Using video diffusion to improve sparse-view 3D reconstruction is not new, as shown by 3DGS-Enhancer[1], GenFusion[2], FlowR[3], and ArtiFixer[4]. I do not see a fundamental difference between GeoVerse and these methods.

  2. The related work is incomplete. I suggest the authors include more studies that use video diffusion to improve 3D reconstruction. Although GeoVerse follows a feed-forward approach, optimization-based methods are also relevant. Comparing reconstruction quality and runtime would help demonstrate the advantages of the feed-forward approach.

  3. The proposed global spatial memory is conceptually similar to maintaining a 3DGS representation in previous methods. The main architecture essentially combines ControlNet, GLD, and video diffusion. I see these as engineering improvements rather than methodological innovations. It is also unclear what new problem this combination solves, since improvements from introducing a stronger pretrained video diffusion model into GLD are largely expected.

  4. The paper lacks comparisons with several closely related SOTA methods, including ArtiFixer, FlowR, and Difix3D+[5]. These comparisons are important for evaluating the actual improvements over existing approaches.

  5. The paper also lacks comparisons with recent SOTA feed-forward NVS methods, such as LargeNVS[6].

  6. Few steps distillation has already been explored by Difix3D+ [5]

[1] Liu, X., Zhou, C., & Huang, S. (2024). 3dgs-enhancer: Enhancing unbounded 3d gaussian splatting with view-consistent 2d diffusion priors. Advances in Neural Information Processing Systems, 37, 133305-133327.

[2] Wu, S., Xu, C., Huang, B., Geiger, A., & Chen, A. (2025, June). Genfusion: Closing the loop between reconstruction and generation via videos. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 6078-6088). IEEE.

[3]Fischer, T., Bulò, S. R., Yang, Y. H., Keetha, N., Porzi, L., Müller, N., ... & Kontschieder, P. (2025, October). Flowr: Flowing from sparse to dense 3d reconstructions. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 27702-27712). IEEE.

[4] De Lutio, R., Fischer, T., Chang, Y. Y., Zhang, Y., Wu, Z., Ren, X., ... & Turki, H. (2026, July). ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Models. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (pp. 1-12).

[5] Wu, J. Z., Zhang, Y., Turki, H., Ren, X., Gao, J., Shou, M. Z., ... & Ling, H. (2025, June). Difix3d+: Improving 3d reconstructions with single-step diffusion models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 26024-26035). IEEE. [6] Szymanowicz, S., Chen, M., Wang, J., Rupprecht, C., & Vedaldi, A. (2026). LagerNVS: Latent geometry for fully neural real-time novel view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15443-15453).

Questions for authors
  1. What is the total runtime of the entire pipeline, including all preprocessing and inference steps?

  2. Could the authors provide comparisons with at least ArtiFixer and LargeNVS?

  3. What are the key technical challenges that GeoVerse actually addresses? In my view, the listed contributions mainly come from integrating existing techniques into the pipeline, rather than introducing new solutions to these challenges.

4 comments

We sincerely thank Reviewer NxPV for the thoughtful feedback on our technical contribution, spatial-memory representation, comparison with video-prior methods, baseline coverage, few-step distillation, and efficiency. These suggestions helped us clarify the distinction from explicit-3D refinement methods, expand the related work and baseline comparisons, and provide a more complete runtime analysis. Below are our point-by-point responses:

W1 :

Much recent video-prior work generates or repairs views in image or video latent space. Building on GLD, GeoVerse instead explores direct generation in a geometric latent space at scale by training across 15 datasets, transferring frozen video features into joint RGB and geometry generation, maintaining spatial memory across rounds, and distilling the geometric generator for few-step inference. Together, these components form a unified NVS framework, distinguishing our paradigm from using video models to repair renderings of an existing explicit 3D reconstruction.

W2 :

Thank you for the suggested references. We have added Appendix F, “Additional Related Works,” covering 3DGS-Enhancer, GenFusion, FlowR, ArtiFixer, Difix3D+, LagerNVS, and relevant feed-forward methods. It reviews generative priors for explicit 3D reconstruction, followed by feed-forward modeling and latent geometry. We will integrate this discussion into the main Related Work section in the final version.

W3 / Q3 :

Our global memory maintains a confidence-bearing colored 3D point cloud to render target-aligned guidance between rounds. We adopt a point-based representation rather than fitting a Gaussian scene renderer, thereby avoiding propagating under-constrained boundary artifacts into the generation conditions. Beyond this memory design, unlike ArtiFixer and Difix3D+ that refine rendered explicit reconstructions, GeoVerse performs efficient multi-view generation directly within a geometric feature space. This efficiency stems from extracting video features once and reusing them throughout geometric denoising, rather than naively appending a video diffusion model to GLD as an additional generative stage.

W4 / W5 / Q2 :

Thank you for suggesting ArtiFixer, Difix3D+, FlowR, and LagerNVS as additional baselines. Since the FlowR weights have not been publicly released and we could not obtain access to the LagerNVS checkpoint, we instead evaluate two contemporary methods, AnyRecon and WorldMirror v2, as alternatives for the latter two. Quantitative results are reported in Table 4, and qualitative comparisons are provided in Appendix Fig. 9. We also discuss FlowR and LagerNVS in Appendix F.

Table 4. Quantitative comparisons with additional baseline methods on multiple public datasets.

DatasetMethodPSNR ↑SSIM ↑LPIPS ↓ATE ↓RPEr ↓RPEt ↓Reproj. ↓MEt3R ↓Time (s) ↓
DL3DVArtiFixer16.830.5610.4670.28417.7610.6460.7220.31546.09
DL3DVDifix3D+14.510.4800.4060.2527.9810.3950.6460.2912.60
DL3DVWorldMirror v211.840.4480.5150.2018.5720.3320.5860.3221.06
DL3DVAnyRecon15.190.4390.4510.26813.6160.5090.6200.22936.50
DL3DVGeoVerse19.610.5980.3390.0280.9420.0600.6490.2619.24
RE10KArtiFixer19.410.7280.3460.0790.9450.1130.6430.21943.59
RE10KDifix3D+16.590.6260.3650.0530.8000.0840.5880.2102.64
RE10KWorldMirror v212.370.5770.4160.0661.0540.1070.5700.1820.83
RE10KAnyRecon17.840.6230.3410.0421.0800.0620.5680.16235.82
RE10KGeoVerse21.290.7560.2750.0350.4370.0650.6310.1799.29
Mip-NeRF360ArtiFixer15.530.4170.5410.66920.3801.2580.7120.29943.48
Mip-NeRF360Difix3D+14.860.3600.4680.40615.0580.8840.6350.2912.52
Mip-NeRF360WorldMirror v213.660.4020.5270.2088.3300.4910.5230.2870.75
Mip-NeRF360AnyRecon16.470.4030.4390.52516.9090.9930.5920.21135.74
Mip-NeRF360GeoVerse20.020.5320.3510.0711.6290.1680.5940.2449.12

W6 :

Difix3D+ adapts SD-Turbo into a single-step diffusion enhancer to correct rendered views. This enhancement operates in a 2D image latent space, after which the corrected views are optimized back into an explicit NeRF or 3DGS representation. In contrast, GeoVerse distills camera-conditioned multi-view flow matching directly within a geometric latent space. During inference, our method performs three L1 updates followed by one shallow-cascade update while reusing frozen video features. Accordingly, our core contribution lies in this geometric-latent formulation and its empirical validation, rather than few-step diffusion in general.

Q1 :

As reported in Table 5, a full end-to-end generation round covering all preprocessing and inference stages takes 9.76 s, with model inference accounting for 9.23 s on a single NVIDIA A800 80GB GPU using two context images and six target views at 504×504 resolution.

Table 5. Total runtime including all data-preprocessing and model-inference stages.

StageTime (s)
Data Preprocess0.53
Model Inference9.23
Total9.76

Thank you to the authors for the additional experiments and discussion of related work. The new results partially address my concerns about performance and baseline comparisons. However, I am still not fully convinced by the responses to W1 and W3/Q3, especially regarding the technical novelty.

I agree that point clouds and Gaussians have their own advantages, but the rebuttal does not clearly explain what is new beyond combining existing techniques. I have three suggestions that I believe would make the paper more convincing and technically solid:

  1. Why does GLD need video diffusion priors? Could similar improvements be achieved by training GLD on more diverse datasets with spatial memory, without using video diffusion features? I think this comparison would help clarify the actual contribution of video diffusion priors.

  2. Why use a ControlNet-like architecture to incorporate video diffusion priors? If the video features contain geometric distortions or camera-pose inconsistencies, can this architecture handle them, or could the conditioning be too strong? I think the authors should better explain why this design is preferred over other ways of using video diffusion priors.

  3. What is new about the spatial memory mechanism? Similar ideas have already been explored in Lyra 2.0, and I would like to see a clearer discussion of what makes the proposed design different.

Overall, I believe that understanding why and how video diffusion priors improve GLD is more valuable than performance gains alone. The additional experiments strengthen the paper, and I recognize its contributions in terms of performance and exploring how to combine GLD with video diffusion priors. I am therefore willing to raise my score to 3, although I still have some concerns about the technical novelty.

We sincerely thank the reviewer for raising the score, as well as for the thoughtful follow-up questions. We alsoapologize for our delayed response.

Q1. Why does GLD need video diffusion priors?

Across ablation variants trained on the same 15-dataset pool, scaling training data improves both in-domain and out-of-domain performance, while incorporating the Wan prior yields an even larger gain. This improvement is especially pronounced on the out-of-domain ScanNetv2 dataset, where multi-round generation progressively exposes unobserved regions and heightens reliance on a strong generative prior. Table 1 further disentangles the contributions of the video prior, target hints, injection design, and training losses on DL3DV.

Table1. Ablation comparison on the DL3DV dataset.

VariantPSNR ↑SSIM ↑LPIPS ↓ATE ↓RPEr ↓RPEt ↓Reproj. ↓MEt3R ↓
GeoVerse19.610.5980.3390.0280.9420.0600.6490.261
w/o Wan2.2 prior18.220.5550.3670.0451.2040.1090.6530.263
w/o target hint18.590.5630.3710.0401.1150.0950.6510.260
MoT video-KV18.190.5520.3800.0471.2290.1160.6560.268
w/o RGB loss19.150.5890.3540.0321.0190.0610.6500.260
with depth loss19.190.5950.3500.0331.0390.0690.6500.262

Q2.1. Why use a ControlNet-style injection architecture?

We compared ControlNet-style residual injection with a simpler MoT video-KV alternative, where geometric queries attend to cached Wan keys and values. As shown in Table 1, MoT provides no clear improvement, while the ControlNet-style design performs substantially better. Both Wan and geometric-backbone features are dense, spatially aligned image features, making multi-scale residual injection more suitable for transferring appearance priors. Zero-initialized projections also preserve the pretrained denoiser and allow it to learn the appropriate conditioning strength.

Q2.2. Can distorted or pose-inconsistent video features over-condition the model?

We treat Wan features as learned appearance priors rather than exact geometric constraints. Target-aligned RGB-D hints and visibility masks condition both branches, while the camera-conditioned geometric denoiser anchors known regions and uses the video prior primarily to complete unobserved content. Learned residual adapters can further suppress inconsistent cues.

Q3. What is new about the spatial memory compared with Lyra 2.0?

We agree that GeoVerse and Lyra 2.0 share the general idea of persistent 3D memory. Lyra 2.0 stores independent per-frame depth maps and point clouds for retrieval and correspondence, while keeping its final 3DGS reconstruction separate. GeoVerse instead maintains a unified colored point cloud, uses predicted confidence for overlap-aware updates, and renders target-aligned RGB-D hints for subsequent rounds.

We therefore do not claim persistent memory alone as the novelty. Its contribution lies in closing the generation loop within a unified NVS framework that combines geometric-latent generation, video-prior transfer, confidence-aware memory updates, and few-step inference for long-sequence and revisit-consistent synthesis.

Reviewer_29rZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoVerse for long-sequence nvs. It combines video generation priors with latent diffusion to improve image quality and geometric consistency. It uses a global 3D spatial memory to guide new view generation and a distillation strategy to speed up inference.

Critical strengths
  1. Combining video generation priors with geometric latent diffusion is an interesting idea.

  2. The method shows better results on several benchmarks, with improved visual quality and geometric accuracy.

Critical weaknesses
  1. Unclear error accumulation. The method repeatedly adds generated rgbd results to its spatial memory. Errors from earlier rounds may affect later predictions. It is unclear how the method prevents or corrects these errors.

  2. Lack of long-term consistency analysis. The paper does not show how performance changes as the num of generation rounds increases. It is unclear whether the method can build a globally consistent scene over long trajectories.

  3. Unclear pipeline illustration. Fig 2 is crowded and difficult to follow. In particular, the Wan2.2 VACE block has no clear output connection to the geometric latent diffusion block, making the feature injection process hard to understand.

Questions for authors
  1. How should the ability to explore a scene be evaluated? Can the authors provide metrics for newly explored scene coverage, rather than relying mainly on novel view quality?

  2. How does the method prevent incorrect generated geometry or appearance from accumulating in the spatial memory?

  3. Can the authors test a loop trajectory that returns to previously explored regions to verify global scene consistency?

5 comments

We sincerely thank Reviewer 29rZ for the constructive comments on error accumulation, long-horizon consistency, exploration evaluation, and method presentation. These comments motivated us to strengthen the robustness analysis, add multi-round and revisit evidence, and clarify the feature-injection diagram. Below are our point-by-point responses:

W1 / Q2 :

Appendix Fig. 7(a) shows that generation retains coherent appearance under target-hint perturbations. Our outputs also include depth confidence (Fig. 7(b)), which can guide whether new observations replace overlapping memory entries. Confidence-gated memory refresh provides a way to revise earlier geometry or appearance instead of permanently trusting every generated observation.

W2 / Q3 :

Table 1 reports performance across successive generation rounds. While baseline methods suffer severe degradation as generation progresses, GeoVerse with spatial-memory updates maintains higher visual fidelity. Visually, Appendix Fig. 8 demonstrates coherent scene appearance and layout across five trajectories spanning six generation rounds, including revisits to previously explored regions. Complementing these qualitative examples, Table 2 further quantifies consistency under exact-camera revisits.

Table 1. Performance over successive generation rounds on the ScanNetv2 dataset (PSNR ↑ / SSIM ↑ / LPIPS ↓).

MethodRound 1Round 2Round 3
GLD20.42 / 0.773 / 0.36014.92 / 0.622 / 0.49312.62 / 0.563 / 0.556
ViewCrafter16.82 / 0.716 / 0.48912.07 / 0.571 / 0.63210.26 / 0.492 / 0.709
NeoVerse16.50 / 0.639 / 0.49712.82 / 0.484 / 0.59110.38 / 0.426 / 0.646
Gen3C14.98 / 0.635 / 0.48113.64 / 0.578 / 0.52012.31 / 0.553 / 0.597
Ours22.81 / 0.876 / 0.30618.73 / 0.762 / 0.42217.41 / 0.705 / 0.476

Table 2. Exact-camera revisit consistency on the five scenes in Fig. 8.

CasePSNR ↑SSIM ↑LPIPS ↓
A31.090.9570.056
B28.200.9410.094
C27.550.9200.118
D32.270.9600.064
E27.440.9300.135
Mean29.310.9420.093

W3 :

Thank you for identifying this ambiguity. The previous method diagram did not make the output connection between Wan2.2 VACE and geometric latent diffusion sufficiently explicit. We revised Fig. 2 to show the extracted video features, their projection adapters, and their connections to geometric denoising.

Q1 :

Across reference-valid target pixels, initially observed and unobserved regions account for 63.19% / 36.81% on DL3DV, 75.40% / 24.60% on RE10K, and 62.32% / 37.68% on Mip-NeRF360, respectively. Given these substantial proportions, we evaluate reconstruction fidelity on observed regions using PSNR/SSIM and generation quality on unobserved regions using FID/KID, as reported in Table 3. This region-wise protocol cleanly disentangles reconstruction from generation capability, with GeoVerse achieving the best performance across both aspects and all three datasets.

Table 3. Quantitative comparison of fidelity in reconstructed regions and quality in generated regions. Recon. reports PSNR ↑ / SSIM ↑; Gen. reports FID ↓ / KID ↓.

MethodDL3DV Recon.DL3DV Gen.RE10K Recon.RE10K Gen.Mip-NeRF360 Recon.Mip-NeRF360 Gen.
GLD19.51 / 0.610197.8 / 0.03522.67 / 0.766199.7 / 0.08819.05 / 0.433246.3 / 0.058
ViewCrafter17.15 / 0.565237.4 / 0.07317.74 / 0.672282.2 / 0.21516.81 / 0.403344.8 / 0.137
NeoVerse12.55 / 0.365210.7 / 0.04612.61 / 0.477224.9 / 0.12613.77 / 0.302271.6 / 0.076
GEN3C18.57 / 0.622263.1 / 0.09920.91 / 0.756240.6 / 0.16818.29 / 0.452326.7 / 0.117
MVGenMaster19.48 / 0.649185.9 / 0.02522.84 / 0.780261.8 / 0.16218.27 / 0.433249.2 / 0.055
Matrix3D14.42 / 0.446243.8 / 0.04617.61 / 0.605385.0 / 0.43016.13 / 0.341259.7 / 0.092
CAMEO11.26 / 0.400255.8 / 0.07813.91 / 0.523217.7 / 0.09811.44 / 0.261395.1 / 0.201
Ours21.06 / 0.654142.3 / 0.02024.27 / 0.806148.0 / 0.05620.53 / 0.481201.2 / 0.045

Thank you for the response!

I still have concerns aligning with the weaknesses I mentioned in the initial review:

  1. Error accumulation. I checked Fig 7 in the supp and I understand the pipeline you proposed. But Fig 7 cannot prove that your proposed method can alleviate or solve error accumulation issue. Only experiments would tell. Since the output novel views are directly obtained from diffusion model, i recommend to conduct experiments for your method and other baselines, testing how's the psnr goes when we generate 200, 500, and 1000 frames.

  2. Long term consistency analysis. I'm aware that this method generates novel views many rounds, but we have no idea how long is one round. Besides, how can we define long in this task? I think the number of generated frames can present how 'long' the generative scene is. Besides, Since you also claim in the rebuttal that GeoVerse has the capacity of spatial-memory updates, it would be better if you can conduct experiments for this quantitatively. For instance, you can conduct experiments on 3d scene recon dataset, eg, MipNerf360, or datasets shown in fig 9 in the supp material. Given a starting camera pose, how's the psnr when the camera turn 360 degree around? Would the generated frame the same as the initial one when you turn around and go back to the initial camera pose? Only this kind of experiments can prove the efficiency of your long term consistency and the capacity of designed spatial memory updates.

  3. Appreciate for authors answer about Q1. Thank you!

Generally speaking, only text explanation cannot prove how does this method solve the error accumulation issue/long term consistency concern, or is the proposed method better than other baselines regarding those concerns. Only comprehensive quantitative comparisons would tell.

I'm open to discussion. Thank you again for the discussion!

Thank you for your thoughtful follow-up and for suggesting concrete tests of error accumulation and revisit consistency.

Q1. Quality across generation rounds.

We present a case study on the Mip-NeRF 360 room scene featuring a 360-degree trajectory that loops back near the initial camera pose. Each inference group consists of three context views and five target views (3C+5T). For the video baselines, we insert 32 interpolated camera poses between consecutive target views to smooth camera motion and stabilize generation. In contrast, geometry-latent methods directly infer each 3C+5T group and support large viewpoint shifts. Consequently, sequence length is not determined by the number of generated frames alone, since trajectory coverage and viewpoint displacement also play a key role. Appendix Figure 12 illustrates four groups along this trajectory, and Table 1 reports the mean full-image PSNR against the ground-truth photographs for the two displayed targets in each group.

Table 1. PSNR (dB) on the displayed target views.

MethodRound 1 ↑Round 2 ↑Round 3 ↑Round 4 ↑
Gen3C17.1813.9911.6913.09
NeoVerse13.3512.6610.999.97
ViewCrafter16.0716.6814.6212.95
GLD16.1617.0812.8812.53
Ours18.6319.4816.9914.63

Q2. Consistency on revisit.

Appendix Figure 13 illustrates the qualitative results when the trajectory loops back near the initial camera pose. Although the four baselines generate the revisit targets directly from the original reference views, all of them still suffer from noticeable inconsistencies between the initial and revisited predictions. While methods equipped with memory or history conditioning, such as ViewCrafter and NeoVerse, alleviate this discrepancy to some extent, their underlying video generators do not explicitly enforce multiview consistency, leaving visible content drift upon revisit.

To evaluate this behavior quantitatively, Table 2 measures two complementary properties, namely revisit accuracy and revisit consistency. Specifically, revisit accuracy compares each full predicted image against the ground-truth photograph at the same camera pose, whereas revisit consistency evaluates the agreement between the initial and revisited predictions over their shared visible regions after geometric reprojection.

Table 2. Accuracy and consistency on the two revisit pairs

MethodRevisit PSNR vs. GT ↑Revisit LPIPS vs. GT ↓Overlap PSNR ↑Overlap LPIPS ↓
Gen3C17.240.45018.690.413
NeoVerse12.960.60316.610.433
ViewCrafter16.620.49820.470.379
GLD16.910.43719.930.379
Ours20.940.31921.410.324

Across these revisit pairs, GeoVerse achieves the highest revisit accuracy by matching the ground-truth photographs most closely, while also best preserving its earlier predictions under both overlap consistency metrics. Overall, these results quantify long-sequence generation fidelity and demonstrate the effectiveness of our spatial memory in maintaining multiview consistency upon loop closure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.