- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a method called Vid2Orbit, which adapts a pre-trained video extension model to a sparse multi-view generation model, aiming to improve the quality of 3D scene reconstruction from a single image. The authors point out that traditional video image processing temporal compression mechanisms lose important details when the viewpoint changes significantly. Therefore, this paper proposes a camera alignment feature interaction module based on a shared 3D feature field. This module improves multi-view consistency by resetting the VAE for each viewpoint, assigning independent latent variables to each target camera, jointly generating a complete viewpoint sequence using a video extension transformer, and introducing real geometry monitoring. Experimental results show that this method improves the reconstruction fidelity of the VAE and demonstrates certain advantages in COLMAP point cloud reconstruction and ReSplat visualization results.
Critical strengths
-
This research has a clear motivation and employs a relatively innovative problem-solving approach. This paper focuses on the mismatch between the temporal compression mechanism of video diffusion models and the requirements of sparse multi-view reconstruction. Through VAE reconstruction experiments, we analyze the detail loss caused by temporal compression and propose a per-view resetting strategy. This method is reasonable and effective, providing insights for its application in 3D reconstruction tasks.
-
This method has practical value and achieves good experimental results. This paper combines techniques such as per-view encoding, synchronous multi-view generation, and 3D feature interaction. The generated results can be directly applied to existing reconstruction workflows, such as COLMAP and ReSplat. Experimental results show that the generated viewpoints can provide more effective multi-view matching information, and multiple visualizations also demonstrate good detail preservation and scene consistency, proving the applicability of the proposed method.
Critical weaknesses
-
The actual contribution of the core 3D feature exchange module has not been fully validated. Although the paper lists camera-aligned feature exchange as one of its main innovations, existing experiments have failed to clearly demonstrate its superior performance compared to using RGB adaptation alone. According to Table 2 and Appendix B.5, the original RGB adaptation model achieved 14,197.60 dense points on the Synthetic 20 dataset, while the results dropped to 12,407.10 and 12,355.10 points respectively after introducing the 3D feature exchange branch. While these results are not the result of fully controlled module ablation, they still raise questions about the effectiveness and necessity of this core design, weakening the paper's technical contribution.
-
The quantitative evaluation of 3D reconstruction quality is insufficient, and the fairness of comparative experiments needs improvement. I believe this is the biggest problem I found after reading the paper. This paper mainly uses COLMAP point cloud amount, feature trajectory, and reprojection error to evaluate reconstruction performance, but these metrics do not directly reflect the accuracy of the real geometry. Although this paper uses synthetic data containing real geometric information, there is a lack of direct evaluation of geometric errors or the rendering quality of new perspectives. Furthermore, I believe that this paper has too few quantitative evaluation indicators and the experimental scale is not large enough.
Questions for authors
- Why are the reconstruction metrics on Synthetic Dataset 20 lower than those on the RGB Adaptive Dataset after introducing Camera Adjustment Feature Improper? Can fully modular resection experiments be performed under uniform training conditions to clarify the model configuration in each experiment and verify the independent contribution of shared 3D feature fields?
- Since the number of COLMAP point clouds does not directly represent 3D geometric precision, can true improvements in final 3D reconstruction performance be demonstrated by reporting metrics such as chamfer distance, depth error, and new perspective rendering quality using existing synthetic data?
- Different baseline models maintain the same camera parameters, field of view, and custom scaling. Do these differences affect the COLMAP matching and reconstruction results? How are these factors controlled? Or can more consistent comparative experiments be provided?
- As shown in Table 1, since K=1 and K=4 differ in the number of latent variables, the improvement in PSNR is likely partly attributed to an increase in representation capacity. Can you further analyze whether the view-by-view reconstruction strategy still holds a significant advantage under the same latent variables or computing resource budget?
3 comments
Thank you for the explanation. I believe the basic questions have been answered quite clearly, and the article is of good writing quality. Considering the content of the article and the scores from other reviewers, I will maintain my original score and support the article's acceptance. Good luck!
Response to Reviewer fhwp's Comment
Thank you for your thoughtful review and support. We are pleased that the response has clarified the main questions, and we appreciate your recognition of the paper’s presentation. Your constructive comments have helped us strengthen both the experimental evidence and the explanation of our method.
Response to Reviewer fhwp
Response to Reviewer fhwp
Thank you for the questions on attribution, accuracy, cameras and capacity; we respond with matched training and controlled evaluations.
1. Branch contribution and decreased point count (W1/Q1). The drop spans training stages: Table 2 evaluates the RGB-adapted generator, whereas Tables 3/4 compare ordinary/distilled students that both contain the field and share a later initialization. The Synthetic20 stages are:
Appendix B.5 repeats the RGB checkpoint; Table 4 expands Table 3's pair with sparse-point, track and reprojection statistics. Branch adaptation changes observations and COLMAP's retained correspondences, allowing fewer points with lower retained reprojection error; accuracy therefore requires direct surface evaluation. Within the teacher pair, Real20 gains 12.39% dense points and reprojection error falls from 1.425 to 1.357 px, while Synthetic20 dense yield changes by −0.42%.
For your matched-training ablation (Appendix C.3), G0/F/G1 share one RGB initialization, data, 2,000 updates, seed 73, backbone, optimizer and camera/codec settings without a teacher, using the same full-orbit person evaluation:
G1 lowers mean person AbsRel/Chamfer by 14.6%/16.4% versus RGB-only G0 and Chamfer by 15.2% versus F, isolating a supervision benefit with the F/G1 architecture fixed. Figures 7/9 visualize the learned G1 surface structure.
2. Direct image and reconstruction accuracy (W2/Q2). For your depth, surface-distance and rendering questions, complete distilled Vid2Orbit and baselines use common cameras on additional synthetic data (Appendix C.2). Identical fixed-camera ReSplat uses 16 generated views per method, testing held-out ±9°/±27° views on a shared GT-mutually-visible person region:
Relative to the best baseline for each metric, render PSNR rises 4.34 dB and mean AbsRel/Chamfer fall 65.9%/54.8%. These gains show more accurate observed-person geometry and novel-view rendering under the same reconstruction procedure. RGB uses the full GT region; depth requires opacity ≥0.95; symmetric mean Euclidean Chamfer uses no ICP/GT scale fitting. Scores average valid views then scenes equally.
Direct RGB uses 19 non-source targets; person PSNR averages RGB errors inside nonempty GT person masks, while other metrics use full images:
Full-orbit PSNR penalizes both generation errors and plausible unseen-content completions that differ from the single GT. Against the best baseline per metric, Vid2Orbit gains 1.27 dB full-image and 3.40 dB person PSNR and reduces LPIPS by 6.5%, supporting better target-view fidelity in both domains.
3. Cameras, FOV and scale (Q3). To control camera/FOV confounds, tests use common requested intrinsics/poses and lossless 832×480 outputs; baseline scale uses predicted geometry without target-GT fitting. Frames 0,4,...,76 of 81-frame outputs select the requested positions without the repeated endpoint, and direct scoring excludes the source. Fixed cameras prevent pose optimization from absorbing generation errors, complementing the original native-camera tables (Appendix B.7).
4. Equal latent capacity and codec cost (Q4). We fix VAE weights and latent dimensions (Section 4.2; Appendix C.6): K=1 encodes 10/20 requested views and K=4 encodes 37/77 dense GT frames into the same budgets, scoring identical target cameras:
K=1 gains 5.59/5.38 dB and wins in every evaluated scene within each budget, demonstrating better view fidelity without increasing latent capacity. With no diffusion sampling in this test, the improvement is attributable to the codec, supporting per-view reset for sparse reconstruction. For measured cost at 20 slices:
K=1 reduces measured codec time by 73.4% and peak allocated memory by 60.4%, combining higher fidelity with lower codec cost in these implementations. Time includes encode/decode and CPU transfer, memory includes resident evaluation networks, and loading/warm-up are excluded; these are codec-stage measurements, not end-to-end generator costs.