acceptodds
Under review as a conference paper at ICLR 2027

OrbitNVS: Harnessing Video Diffusion Priors for Novel View Synthesis

Abstract

Novel View Synthesis (NVS) aims to generate unseen views of a 3D object given a limited number of known views. Existing methods often struggle to synthesize plausible views for unobserved regions, particularly under single-view input, and still face challenges in maintaining geometry- and appearance-consistency. To address these issues, we propose OrbitNVS, which reformulates NVS as an orbit video generation task. Through tailored model design and training strategies, we adapt a pre-trained video generation model to the NVS task, leveraging its rich visual priors to achieve high-quality view synthesis. Specifically, we incorporate camera adapters into the video model to enable accurate camera control. To enhance two key properties of 3D objects, geometry and appearance, we design a normal map generation branch and use normal map features to guide the synthesis of the target views via attention mechanism, thereby improving geometric consistency. Moreover, we apply a pixel-space supervision to alleviate blurry appearance caused by spatial compression in the latent space. Extensive experiments show that OrbitNVS significantly outperforms previous methods on the GSO and OmniObject3D benchmarks, especially in the challenging single-view setting (eg, +2.9 dB and +2.4 dB PSNR).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel View Synthesis, Video Diffusion Models, Normal Map Generation, Multi-View Consistency, 3D Vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YwpEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OrbitNVS addresses object-centric novel view synthesis from one or a few reference images by adapting Wan2.1-I2V-14B to generate RGB and normal-map sequences along specified camera trajectories. The method combines camera adapters conditioned on Plücker ray embeddings, a parameter-sharing normal branch that interacts with the RGB branch through self-attention, and two-stage training that adds decoder-space pixel supervision after latent-space flow-matching training. The reported results outperform SV3D, SEVA, and EscherNet in PSNR, with the largest advantages in the single-reference setting.

Critical strengths
  1. Substantial and consistent improvements in single-reference object-centric NVS. OrbitNVS achieves the highest reported PSNR across both GSO and OmniObject3D under all three evaluated camera trajectories. For level orbits, it improves over the strongest PSNR baseline by, and retains an advantage as elevation variation increases. The magnitude and consistency of these gains constitute strong evidence of the method’s effectiveness in its stated setting.
  2. Useful empirical evidence for the normal branch and pixel-space training. Removing the normal branch reduces PSNR by 0.9–1.4 dB across the six single-reference settings, while removing pixel supervision reduces it by 0.2–1.0 dB. The pixel-loss comparison controls the total number of training steps, making the evidence for this training objective more persuasive.
Critical weaknesses
  1. Insufficient comparison with related baselines.
  • Lack direct comparisons against RGB/normal generation approach including Wonder3D [1], Wonder3D++ [2], Era3D [3]. These methods provide public implementations and weights for multi-view RGB/normal generation and their downstream mesh reconstruction does not preclude comparing the preceding generation stage.
  • Methods like MVGenMaster [4] also offer a relevant RGB NVS baseline with variable reference views and camera conditioning
  • Image-to-3D system including Hunyuan3D [5] or Trellis [6] could additionally be evaluated by rendering its generated assets at the test viewpoints
  1. Key methodological details and the pixel-supervision design remain insufficiently specified.
  • Cross-branch attention: The exact operation that exchanges information between the RGB and normal branches is not specified.
  • Reference masks: It is unclear how frame-level reference masks align with temporally compressed VAE latents, particularly when known and unknown frames share a compressed temporal unit
  • Pixel-loss target: Equation 7 compares decoded noisy intermediate states, but the choice of this target over clean-image supervision is not justified.
  • Optimization details: The paper does not clearly specify whether the VAE decoder is frozen or how the pixel and latent losses are weighted. I assume VAE decoder should be frozen while training DiT, but the authors are recommended to clearly specify it and justify how is it different from training latent loss alone theoretically.
  1. The experimental scope limits the broader conclusions about NVS. Quantitative validation is confined to rendered object-centric benchmarks, and the real-photograph demonstrations use background-removed inputs. There is no scene-level evaluation testing substantial depth variation, background structure, and camera translation through an environment. Object-centric evaluation is appropriate for the stated task, but it does not establish the broader applicability of the adaptation strategy or general superiority over methods such as SEVA. Scene-level evidence, or a more consistently restricted framing of the conclusions, is needed to substantiate the broader significance of the work.

[1] Long, Xiaoxiao, et al. "Wonder3d: Single image to 3d using cross-domain diffusion." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Yang, Yuxiao, et al. "Wonder3d++: Cross-domain diffusion for high-fidelity 3D generation from a single image." IEEE Transactions on Pattern Analysis and Machine Intelligence 48.2 (2025): 1674-1688.

[3] Li, Peng, et al. "Era3d: High-resolution multiview diffusion using efficient row-wise attention." Advances in Neural Information Processing Systems 37 (2024): 55975-56000.

[4] Cao, Chenjie, et al. "Mvgenmaster: Scaling multi-view generation from any image via 3d priors enhanced diffusion model." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[5] Hunyuan3D, Team, et al. "Hunyuan3d 2.1: From images to high-fidelity 3d assets with production-ready pbr material." arXiv preprint arXiv:2506.15442 (2025).

[6] Xiang, Jianfeng, et al. "Native and compact structured latents for 3d generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

Questions for authors
  1. Clarification — cross-branch interaction. What exact self-attention operation connects the RGB and normal branches? Please specify how their tokens and Q/K/V are organized and which parameters are shared.
  2. Clarification — reference conditioning. How are frame-level reference masks represented after temporal VAE compression? A tensor-shape example with one known frame and several unknown frames in the same compressed temporal unit would clarify whether individual reference positions are preserved.
  3. Clarification — pixel supervision. Does the implementation follow Equation 7 literally, comparing decoded noisy intermediate states? If so, why choose this target over clean-image supervision, is the decoder frozen, and how are the pixel and latent losses weighted?
  4. Clarification — evaluation protocol. Given the different output resolutions and frame counts, how are target camera poses and image resolutions matched across methods when computing metrics? Are reference views excluded, and is the same foreground-masking policy applied?
  5. Additional evidence — relevant baselines. Can you provide a compatible comparison with a joint RGB/normal generator, such as Wonder3D++ or Era3D, and a recent camera-conditioned NVS method, such as MVGenMaster? These comparisons would help establish whether the reported advantage extends to the closest alternative approaches.
  6. Additional evidence — quality versus cost. How does OrbitNVS compare with the evaluated baselines in runtime and peak memory under matched hardware and output settings? Does the normal branch retain a quality advantage over the RGB-only variant at a matched inference-time budget? Future-work suggestion: Scene-level evaluation would be valuable, but is not a prerequisite if the paper consistently restricts its claims to object-centric NVS.
Reviewer_unzYOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper fine-tunes a video diffusion model for novel view synthesis (NVS) and achieves good results on the GSO and OmniObject3D benchmarks.

Critical strengths
  1. The paper is clearly written and easy to understand.

  2. The proposed normal map supervision and pixel-space loss appear to work well.

Critical weaknesses
  1. Leveraging video diffusion priors for novel view synthesis is not new, from 3DGS-Enhancer [1] to GenFusion [2], FlowR [3], and ArtiFixer [4]. However, the authors completely ignore these highly related works. In my view, the core novelty does not exist. Even though this paper only uses video diffusion for NVS without involving 3D reconstruction, its objective is the same as the papers I mentioned: leveraging video diffusion priors for NVS.

  2. The core architecture of this paper is not fundamentally different from CAT3D or SEVA. The former uses image diffusion, while this paper uses video diffusion. However, previous methods also incorporate video diffusion mechanisms, such as 3D convolution or cross-frame attention, to concatenate features from multiple frames. I do not think this paper introduces any fundamental architectural difference.

  3. The paper does not compare with current state-of-the-art methods, such as ArtiFixer and LargerNVS [5]. Moreover, the authors do not provide any out-of-distribution comparisons. If the proposed method really works, I believe it is necessary to evaluate it on more diverse scenes, such as the outdoor scenes in DL3DV.

  4. RGB loss and normal loss have already been explored in previous papers [6] that leverage diffusion models for depth estimation.

[1] Liu, X., Zhou, C., & Huang, S. (2024). 3dgs-enhancer: Enhancing unbounded 3d gaussian splatting with view-consistent 2d diffusion priors. Advances in Neural Information Processing Systems, 37, 133305-133327.

[2] Wu, S., Xu, C., Huang, B., Geiger, A., & Chen, A. (2025, June). Genfusion: Closing the loop between reconstruction and generation via videos. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 6078-6088). IEEE.

[3]Fischer, T., Bulò, S. R., Yang, Y. H., Keetha, N., Porzi, L., Müller, N., ... & Kontschieder, P. (2025, October). Flowr: Flowing from sparse to dense 3d reconstructions. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 27702-27712). IEEE.

[4] De Lutio, R., Fischer, T., Chang, Y. Y., Zhang, Y., Wu, Z., Ren, X., ... & Turki, H. (2026, July). ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Models. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (pp. 1-12).

[5] Szymanowicz, S., Chen, M., Wang, J., Rupprecht, C., & Vedaldi, A. (2026). LagerNVS: Latent geometry for fully neural real-time novel view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15443-15453).

[6] He, J., Li, H., Yin, W., Liang, Y., Li, L., Zhou, K., ... & Chen, Y. (2025, May). Lotus: Diffusion-based visual foundation model for high-quality dense prediction. In International Conference on Learning Representations (Vol. 2025, pp. 89454-89467).

Questions for authors

Can you provide the comparison with LargeNVS, FlowR and artifixer in DL3DV dataset?

Reviewer_icTzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces OrbitNVS, which is a method for single (or very sparse) Novel View Synthesis (NVS). The paper utilises a camera adapter to condition the model on the orbital trajectory and generate a full set of images. To improve the geometrical properties of the rendered objects, the authors propose to include a normal rendering branch in the approach. The work includes experiments on single- and up to 3-view input NVS.

Critical strengths

The main strength of the model is the strong performance on the reported baselines. The presented samples, as well as the downstream renderings from trained 3DGS models, look convincing. It also seems that the model is robust to elevation changes.

I find the idea of sharing the RGB generation task with normal generation to be sound and convincing for generating better geometry.

Critical weaknesses

I am not fully convinced about the novelty of the paper. The camera adapters were introduced in CameraCtrl, joint normal prediction in Wonder3Dm and pixel-space post-training in "Pixel-space post-training of latent diffusion models". Whilst a clever combination of ideas from other papers is not in itself grounds for rejecting the novelty claim, the paper does not go into many details of what is novel in this combination.

The next worry is that the performance may just come from extremely strong backbone and much larger complexity of the model. Wan is a model of a tenfold complexity compared to other baselines, which are smaller and trained from much weaker priors. Similarly, the training set is created with exactly the same trajectories as used for testing. This both constrains the task heavily and casts some doubt on the generalisation capability. Further, the PSNR difference is not huge between OrbitNVS with variants excluding normals or pixel-loss (should also include w/o both); SSIM and LPIPS differ even less - it would be great to investigate performance vs computational overhead for this. In Appendix A.1, the authors touch on the prior importance by stating that CogVideoX-5B loses only 0.43dB as a prior; this could include the full table and breakdown.

Questions for authors
  • The generation time is a bit hidden in C.2. It would be good to show a comparison of generation time against the other baselines.
  • The qualitative samples suggest that the approach recovers detail of the backfaces of the objects very well. This could spark a question about what kind of objects were seen in the Wan training set.
  • Why is the generation constrained to an orbital video? How would a novel trajectory perform (C.3 does not say much)?
  • The 3DGS samples are included; it would be great to see similar ones from other compared methods as well.
  • How do the results look if we keep increasing the number of input views?
  • Interestingly, the authors mention that training on the whole Objaverse dataset degrades the performance. The whole Objaverse is what the other baselines trained on, so including this in a comparison could give an overview of how important the filtering of the assets is.
  • It would be worth including 'fixers' in the family of diffusion-guided NVS-related work: Difix3D+ [1], ViDAR [2], ArtiFixer [3].

[1] Jay Zhangjie Wu, Yuxuan Zhang, Haithem Turki, Xuanchi Ren, Jun Gao, Mike Zheng Shou, Sanja Fidler, Zan Gojcic, Huan Ling. Difix3D+: Improving 3D reconstructions with single-step diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025 [2] Michal Nazarczuk, Sibi Catley-Chandar, Thomas Tanay, Zhensong Zhang, Gregory Slabaugh, Eduardo Pérez-Pellitero. ViDAR: Video diffusion-aware 4D reconstruction from monocular inputs. In Advances in Neural Information Processing Systems (NeurIPS), 2025. [3] Riccardo de Lutio, Tobias Fischer, Yen-Yu Chang, Yuxuan Zhang, Jay Zhangjie Wu, Xuanchi Ren, Tianchang Shen, Katarina Tothova, Zan Gojcic, Haithem Turki. ArtiFixer: Enhancing and extending 3D reconstruction with auto-regressive diffusion models. In ACM SIGGRAPH 2026 Conference Papers, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.