acceptodds
Under review as a conference paper at ICLR 2027

Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis

Abstract

We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code will be released publicly.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel View Synthesis, Visual Geometry, Diffusion Models, Geometric Consistency

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_1FK1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes a geometry-routed multi-view diffusion model for sparse-view novel view synthesis, which combines the advantages of both reconstruction-based NVS methods and diffusion-based NVS methods. It presents three key parts:

(1) Geometry-routed generation: a multi-view diffusion framework whose confidence-aware Visual Geometry Router (VGR) transforms VGGT-Ω features and 3D locations into query-aligned conditions, grounding generative completion in observed scene structure.

(2) Geometry-grounded consistency: Point-Track Residual Consistency (PTRC), which aligns predicted-clean residuals along reliable 3D correspondences to reduce cross-view drift without suppressing valid view-dependent appearance.

(3) Geometry-condition regularization: stochastically attenuating or dropping routed geometry during training to prevent over-reliance on imperfect projections, improving robustness under sparse or inaccurate geometry.

Additionally, the experimental results have proved the effectiveness of these proposed modules.

Critical strengths
  • This work is well-organized, making it easy to follow.
  • The proposed method is intuitive and interesting, which integrates the advantages of both reconstruction-based NVS methods and diffusion-based NVS methods. It explicitly uses the reconstruction priors to guide the novel view synthesis. In detail, it contains three parts: Geometry-routed generation to integrate the VGGT-Omega features, Geometry-grounded consistency to maintain the multi-view consistency, Geometry-condition regularization to reduce the negative effects of projection noises.
  • This paper has presented sufficient experimental results, measuring the effectiveness of the proposed method.
Critical weaknesses

The paper lacks separate front-only and back-only ablations of the residual refinement branch with the hard anchor retained. Consequently, it remains unclear whether the back hypothesis and front hypothesis provide complementary information.

Questions for authors

Projection errors and feature aggregation in Visual Geometry Conditioning may cause misalignment, blurring, or loss of fine details, particularly near boundaries and thin structures.

Reviewer_DJorOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes VGGT-Diff, which fine-tunes a pretrained video diffusion model (Wan2.1-I2V-14B) for sparse-view novel view synthesis. It routes frozen VGGT-Ω features into each target view through their predicted 3D points, using a z-buffered hard anchor refined by soft front/back depth layers. A training-time Point-Track Residual Consistency (PTRC) loss aligns the posterior mean (x zero) prediction residuals of jointly denoised target views along shared 3D points. Stochastic dropout of the geometry condition enables geometry-only CFG at inference. Trained on ~1K DL3DV scenes, the method reports state-of-the-art PSNR and perceptual metrics on DL3DV and competitive zero-shot results on Mip-NeRF 360.

Critical strengths
  1. The paper is well-written and easy to follow, and the appendix thoroughly specifies the camera gauge, view sampling, pose-difficulty bins and hyperparameters.

  2. The method is carefully engineered. PTRC aligns residuals relative to each view’s own flow target, so it shares its optimum with flow matching across noise levels. Condition dropout and the early-step, target-only Geometry-Prior CFG strengthen geometry without discarding source evidence.

  3. Performance against FrameCrafter, the most comparable baseline (same backbone, similar training scale), is clearly better: +0.92 dB PSNR and +0.063 SSIM on DL3DV. The paired scene-level analysis (92% scene win rate, A.9) shows the gain is consistent rather than driven by a few scenes.

  4. The results are strong and the evaluation is broad. Pose-stratified results show larger margins on far and extrapolated views, and cross-view consistency is verified with two independent reconstructors.

Critical weaknesses

1. The geometry routing seems to be a bit heuristic. The front/back hypotheses rely on hand-set relative-depth thresholds (τ, δ) and a fixed two-layer structure. There is no sensitivity analysis or evidence that the refiner acts at occlusion boundaries as claimed. Given its 0.13 dB contribution (Table 3), the added complexity is not well justified.

2. The advantage of VGGT-Ω features over their pointmap outputs is weakly established. Prior works such as GenWarp (Seo et al., NeurIPS 2024) and MoAI (Kwak et al., ICLR 2026) show that geometric anchors from depth or pointmaps alone suffice for geometry-grounded NVS. The paper’s only related control replaces features with point-rendered RGB, which does not test conditioning on the pointmaps themselves. The small spread across feature layers (0.15 dB, Table 8) further suggests the features may add little beyond the geometry used to route them.

3. The soundness of PTRC is unclear. Because it penalizes the difference of residuals, PTRC is invariant to errors shared across views and can be lowered by correlating errors rather than reducing them. The residuals also live in VAE latent space, where each cell nonlinearly encodes a neighborhood seen at different scale and context in each view. Matching latent errors therefore need not imply consistent pixel- or 3D-level errors. The only control (velocity matching, A.8) imposes an incorrect constraint and does not isolate the effect of cross-view coupling.

4. The performance gains, especially in the ablations, are marginal. Layered refinement (0.13 dB), CFG (0.12 dB) and condition regularization (0.30 dB) come from single runs on 192 targets without confidence intervals and may lie within variance. The no-geometry ablation (17.29 dB) also falls below FrameCrafter on the same targets (~18.02 dB), so the 1.73 dB attributed to geometry is measured against a weakened baseline. In aggregate, DL3DV LPIPS is effectively tied with FrameCrafter (0.222 vs. 0.223).

Questions for authors
  1. How sensitive are the results to τ and δ? Can you report a front-only vs. front+back ablation and metrics restricted to occlusion boundaries?

  2. How does routing VGGT-Ω features compare with routing only pointmaps or coordinates within the same pipeline? What do the features contribute beyond geometry?

  3. Can you compare PTRC against an uncoupled per-view residual loss at the same track locations? What is PTRC’s effect on the consistency metrics in Table 4?

  4. Can you provide multiple seeds or confidence intervals for Table 3? Why does the no-geometry variant underperform FrameCrafter?

When these questions are addressed, I would consider raising my score for the paper.

1 comment
Reviewer_DJorOpenReview ↗

Thank you for your rebuttal

I thank the authors for their careful and well-defined rebuttal. The authors have conducted all experiments I have asked of them, and have given a clear explanation for all the questions I had about the paper, especially the questions regarding the auxiliary PTRC loss. Their experiment on pointmap-only conditioning also validates their contribution as well, I believe. As I have promised, I raise my score to 4.

Reviewer_r79LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes VGGT-Diff for sparse-view novel view synthesis. The method uses a frozen VGGT-Ω to extract geometry and visual features, which are projected into target views through a Visual Geometry Router. Depth layering and residual refinement handle projection conflicts, and the resulting features condition on Wan2.1. The authors further introduce Point-Track Residual Consistency (PTRC), which regularizes the reconstruction residuals of predicted clean latents at corresponding points, and Geometry-Prior CFG, which controls the strength of geometric conditioning during inference. The paper evaluates image quality on DL3DV and zero-shot generalization to Mip-NeRF 360, alongside joint generation, multi-view reconstruction metrics, and component ablations.

Critical strengths

1.The effectiveness of the complete system is supported by a broad range of experiments: the evaluation covers in-domain testing, zero-shot transfer, varying viewpoint difficulty, and joint multi-view generation. In Table 1, the method improves PSNR on DL3DV from FrameCrafter’s 17.180 to 18.104. 2.Using visual features from a geometry model directly as generative conditioning has clear practical value: the method transfers not only geometric predictions but also visual features aligned with the target views. In Table 3, the complete model without CFG achieves a PSNR of 19.024, compared with 18.553 when conditioned on point-rendered RGB and 17.294 when Ω information is removed. These comparisons support the value of geometry-aligned features for conditioning generation.

Critical weaknesses
  1. It remains unclear whether PTRC specifically improves cross-view geometric consistency. The PTRC ablation in Table 3 primarily reports image quality, while the geometric evaluation in Table 4 compares complete systems. Appendix A.4 compares joint and independent generation, which also does not isolate the effect of PTRC. These results therefore cannot establish whether the improvements in geometric consistency arise from PTRC, geometric conditioning, or joint attention.
  2. The improvements on Mip-NeRF 360 are not consistent across all metrics. Although the method achieves strong perceptual quality, it does not obtain the best results on every evaluation metric.
Questions for authors

1.Could you remove only PTRC while keeping the training and joint generation settings unchanged, and report both image quality and Pi3 geometric metrics for Joint-8? 2.Both Table 4 and Table 7 in the appendix involve 52 scenes with eight target views per scene, but the reported Chamfer distances for your method are 0.0733 and 0.0483, respectively. What differences in model checkpoints, target-view sets, generation resolution, or evaluation procedures account for this discrepancy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.