acceptodds
Under review as a conference paper at ICLR 2027

IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation

Abstract

Reconstructing coherent 3D geometry and appearance from unposed multi-view images is a fundamental yet challenging problem in computer vision. Most existing visual geometry foundation models predict explicit geometry by regressing pixel-aligned pointmaps, often suffering from redundancy and limited geometric continuity. We propose IVGT, an Implicit Visual Geometry Transformer that implicitly models continuous and coherent geometry from pose-free multi-view images. This formulation learns a continuous neural scene representation in a canonical coordinate system and supports continuous spatial queries at any 3D positions, retrieving local features to predict signed distance (SDF) values and colors using lightweight decoders. It allows direct extraction of continuous and coherent surface geometry, enabling rendering of RGB images, depth maps, and surface normal maps from arbitrary viewpoints. We train IVGT via multi-dataset joint optimization with 2D supervision and 3D geometric regularization. IVGT demonstrates generalization across scenes and achieves strong performance on various tasks, including mesh and point cloud reconstruction, novel view synthesis, depth and surface normal estimation, and camera pose estimation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Pose-free 3D Surface Reconstruction, Feed-Forward Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GepGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a method for reconstructing continuous implicit fields from multi-view images without requiring known camera poses, using a VGGT-like framework. The core idea is straightforward. The method is trained end-to-end, similar to other feed-forward reconstruction approaches, but leverages the VGGT architecture to enable pose-free reconstruction through alternating frame-wise and global self-attention. The proposed method extends this architecture with a volume rendering head that samples view features from the VGGT latents and predicts SDF and color values to render images, similar to existing volume rendering approaches. The renderer is supervised using pixel-space losses, and query points for volume rendering are sampled using the predicted camera poses. The method employs an existing volume rendering approach, VolSDF, for surface reconstruction. Results in the paper shows that the method is significantly better than generalizable surface reconstruction approaches and performs at-par or little bit inferior that per-scene optimization methods.

Critical strengths
  • The paper is well-written and easy to follow, with clear explanations of the methodology and implementation details.
  • The method demonstrates significant improvements in reconstruction quality over previous generalizable surface reconstruction methods.
Critical weaknesses
  • Limited Technical Novelty: Despite being well-written, the paper offers limited technical novelty, as most components of the proposed pipeline are based on existing approaches. Similar to how VGGT employs different prediction heads for camera poses, depth, and point trajectories, this method introduces an additional volume rendering head for surface reconstruction, which appears to be a relatively straightforward extension of the architecture. Furthermore, the other components required for volume rendering are already well-established in the literature, such as feature sampling in PixelNeRF and surface reconstruction using VolSDF. Therefore, I find the overall technical contribution of the paper to be limited.

  • Improvements Over Per-Scene Optimization Methods: Although the method achieves significant improvements over previous generalizable reconstruction approaches, its improvements over per-scene optimization methods are marginal or, in several cases, worse in terms of metrics such as precision, accuracy (Acc), and F-score. Additionally, the paper does not appear to compare against sufficiently fast per-scene optimization baselines. For example, MonoSDF uses an MLP-based architecture, which is generally slow to optimize. However, MonoSDF also has an Instant-NGP variant that could potentially converge much faster than the 22 hours reported in the paper. Therefore, I believe the evaluation would benefit from comparisons with more efficient per-scene optimization baselines.

  • Unclear Contribution of the Proposed Method: It is difficult to determine whether the improvements over previous generalizable reconstruction methods stem from the proposed approach itself or primarily from the stronger pretrained VGGT backbone. The paper does not clearly isolate the contribution of the proposed components from the benefits of using a more powerful backbone architecture.

Questions for authors
  • I would like to see a more fair comparison with fast per-scene optimization baselines like instant-ngp (neuralangelo), 2D gaussian splatting and its varient.

  • I would like to hear from the authors about what is the novelty of their method.

5 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thanks for your thoughtful comments, questions and insightful suggestions. We provide detailed responses below.

W1 & Q2 - Novelty of our method

We thank the reviewer for the comment. The main novelty of IVGT lies in its formulation of implicit visual geometry learning, rather than in individual components. Our central idea is to infer a shared and continuously queryable 3D field from unposed images, extending visual geometry transformers beyond discrete and view-indexed predictions. This representation provides an amodal description of the scene and a common geometric basis for surface extraction, depth and normal rendering, and novel-view synthesis. The contribution therefore concerns what the model learns to represent and how its outputs are geometrically coupled. Feature sampling and SDF-based volume rendering provide established tools for realizing this idea. We position our contribution at the level of the scene representation and learning formulation, rather than claiming these tools as new.

Specifically, VGGT represents geometry with per-pixel points. IVGT changes both what the model represents and how its geometric outputs are decoded. IVGT provides continuous representation of the 3D scene including the occluded spaces, enabling tasks including surface extraction and novel-view rendering. We acknowledge that PixelNeRF and VolSDF establish important foundations for image-conditioned neural fields and SDF-based reconstruction. Multi-view PixelNeRF assumes known relative camera poses, while VolSDF requires per-scene optimization. However, it is not trivial to implement our formulation (generalizable, continous, and consistent scene reconstruction from unposed images) directly with existing techniques. Challenges include noisy estimated poses, consistent geometry requirement, good generalization ability, simultanously apperance and geometry modeling, etc. In order to address them, we have designed and tried various ways of feature aggregation, positional encoding, training strategies, training objectives, etc., to build a model that achieves strong performance on comprehensive tasks, including mesh and point cloud reconstruction, novel view synthesis, depth and surface normal estimation, and camera pose estimation.

W2 & Q1 - Comparison with fast per-scene optimization baselines

Thanks for the suggestion. Among the per-scene optimization baselines in Table 1 (in our paper), MonoSDF is the only one for which IVGT performs better on some metrics and worse on others. Given that the original MonoSDF requires approximately 22 hours of per-scene optimization, we believe IVGT still offers a substantial advantage. Nevertheless, we agree with the reviewer that comparisons with faster per-scene optimization baselines would make the evaluation more complete. As shown in Table (a), we additionally compare IVGT with (1) the Instant-NGP variant described in the MonoSDF paper, (2) Neuralangelo[1], (3) 2DGS[2], and (4) QGS[3]. We also report the average optimization time measured when reproducing each method.

Table (a). Comparison with fast per-scene optimization baselines.

MethodAcc ↓Comp ↓Chamfer ↓Prec ↑Recall ↑F-score ↑time ↑
MonoSDF-grid0.1230.0710.0970.5480.5480.543~6h
Neuralangelo0.2930.1570.2250.1770.2770.205~14h
2DGS0.0690.1410.1050.4850.4340.458~11min
QGS0.0760.1650.1200.4290.2800.338~3h
IVGT(ours)0.0390.0450.0420.7560.6940.723~20s

IVGT achieves substantially better reconstruction quality while requiring much less time than these faster per-scene optimization methods.

[1]Li Z, Müller T, Evans A, et al. Neuralangelo: High-fidelity neural surface reconstruction[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023: 8456-8465.

[2]Huang B, Yu Z, Chen A, et al. 2d gaussian splatting for geometrically accurate radiance fields[C]//ACM SIGGRAPH 2024 conference papers. 2024: 1-11.

[3]Zhang Z, Huang B, Jiang H, et al. Quadratic gaussian splatting: High quality surface reconstruction with second-order geometric primitives[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 28260-28270.

AuthorsOpenReview ↗

Author Response - Part 2

W3 - Isolate the contribution of the proposed method

Thanks for the suggestion. To isolate the contribution of our proposed method, we reevaluate previous generalizable baselines using the poses and geometry predicted by VGGT, as shown in Table (b).

Table (b). Comparison with generalizable baselines adapted to use VGGT-predicted poses and geometry.

MethodAcc ↓Comp ↓Chamfer distance ↓Prec ↑Recall ↑F-score ↑
SimpleRecon+VGGT0.0610.0760.0680.5320.5290.529
FineRecon+VGGT0.0940.0530.0740.5050.6160.552
MeshSplat+VGGT0.0940.1420.1180.2890.3290.306
TriSplat0.0840.1210.1030.3890.5430.453
IVGT(Ours)0.0390.0450.0420.7560.6940.723

We retain TriSplat’s original results because it already uses Pi3 [4], another VGGT-initialized geometry model. We think these results provide further evidence for the contribution of our proposed method.

[4] Wang, Y., Zhou, J., Zhu, H., Chang, W., Zhou, Y., Li, Z., Chen, J., Pang, J., Shen, C., & He, T. (2026). π³: Permutation-Equivariant Visual Geometry Learning. International Conference on Learning Representations (ICLR).

We hope the above response can help address your concerns. We have also made targeted revisions to the manuscript based on your suggestions. We are happy to answer any additional questions you may have.

Dear Reviewer GepG,

Thank you for your thoughtful review and constructive suggestions, which have significantly improved the quality and clarity of our paper. We sincerely hope that our rebuttal has fully addressed all of your concerns. As the discussion period is ending soon, if you need any further clarification or additional information, please do not hesitate to contact us. We truly appreciate your time and effort devoted to reviewing our work.

Best regards, Authors of Submission 12062

Reviewer_sQnGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IVGT, a framework for learning continuous implicit 3D scene representations from unposed multi-view images. Building upon a pretrained VGGT backbone, the method aggregates multi-view image features and employs SDF-based neural rendering to reconstruct geometry and appearance. The problem is well motivated, as existing feed-forward geometry models typically predict discrete, pixel-aligned representations that may suffer from cross-view inconsistencies and surface discontinuities. The experimental evaluations demonstrate promising results across various tasks, particularly mesh reconstruction, while also achieving competitive performance in pointmap reconstruction, novel-view synthesis, depth estimation, and camera pose estimation.

Critical strengths
  1. Well-motivated problem and clear formulation. The paper addresses an important limitation of existing feed-forward geometry reconstruction methods, such as VGGT, which output discrete pixel-aligned representations that do not directly define continuous surfaces. Learning an implicit scene representation is a meaningful direction for improving geometric continuity and supporting downstream surface reconstruction. The proposed method is clearly presented, and the overall framework is easy to follow.

  2. Promising experimental results across multiple reconstruction tasks. The paper provides extensive evaluations covering mesh reconstruction, pointmap reconstruction, novel-view synthesis, depth estimation, and camera pose estimation. In particular, IVGT demonstrates strong mesh reconstruction performance on ScanNet, significantly outperforming other generalizable baselines and approaching the reconstruction quality of per-scene optimization methods. These results suggest that the proposed framework has practical potential for feed-forward surface reconstruction.

Critical weaknesses
  1. The benefit of learning an implicit representation is not sufficiently demonstrated against classical surface reconstruction methods. The proposed method combines a pretrained feed-forward visual geometry model with an SDF-based neural representation. However, it remains unclear to what extent learning the implicit representation provides an advantage over applying existing surface reconstruction techniques to geometry predicted by a strong feed-forward model. For example, one could reconstruct a mesh from VGGT-predicted pointmaps using Screened Poisson Surface Reconstruction, or fuse its predicted depth maps using TSDF fusion. Although the paper compares against other mesh reconstruction methods, such a direct baseline is missing. I believe this comparison is important for establishing the practical advantage and contribution of the proposed implicit formulation.

  2. The computational trade-offs of implicit reconstruction are not fully evaluated. Compared with explicit geometry prediction, the proposed implicit representation requires repeated spatial queries and neural rendering, which may introduce considerable computational overhead. While Table 1 reports the total mesh reconstruction time for different methods, the paper does not provide a detailed runtime analysis of the implicit querying and rendering processes, particularly for novel-view synthesis or varying numbers of input views. A more comprehensive evaluation of rendering latency, memory consumption, and scalability would help establish whether the benefits of the implicit representation justify its additional computational cost.

Questions for authors
  1. Comparison with classical mesh reconstruction: Could the authors provide a comparison between IVGT and standard mesh reconstruction methods applied to VGGT-predicted geometry, such as Screened Poisson Surface Reconstruction or TSDF fusion? Evaluating these pipelines under the same input and benchmark settings would help clarify the advantage of learning an implicit SDF representation over reconstructing a continuous surface from explicit pointmaps. This is the primary concern of the paper and I will consider raising the score if the authors can adequately address it during the rebuttal.

  2. Novel-view synthesis quality: In Table 4, IVGT consistently achieves worse LPIPS scores than AnySplat, despite obtaining competitive or better PSNR and SSIM in several settings. The authors also acknowledge in Appendix D that the proposed SDF-based representation and geometric regularization can suppress fine-grained details. Could the authors provide a more detailed analysis of the observed perceptual quality degradation, particularly in terms of image sharpness and high-frequency details? It would be interesting to understand whether this limitation primarily arises from the implicit representation, geometric regularization, or the appearance modeling and rendering process.

  3. Origin of improvements over VGGT: Tables 2 and 3 show that IVGT improves over VGGT on several pointmap reconstruction and camera pose estimation metrics, although the improvements are not consistent across all datasets. Since these predictions are produced by auxiliary VGGT-style heads rather than directly from the implicit SDF representation, could the authors clarify where these improvements originate? In particular, are they primarily attributable to additional training data and supervision, or does learning the implicit SDF representation improve the shared geometric features? A controlled experiment with VGGT fine-tuned using the same datasets and training budget, but without the implicit representation, would help isolate the contribution of the proposed method.

4 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thanks for your thoughtful comments, questions and insightful suggestions. We provide detailed responses below.

W1 & Q1 - Comparison with classical surface reconstruction methods

Thanks for the suggestion. We use two classical methods, TSDF fusion[1] and Screened Poisson Surface Reconstruction[2], to obtain surfaces from points predicted by VGGT or IVGT. We then compared these surfaces quantitatively with those directly extracted by our method, as shown in Table (a).

Table (a). Comparison with classical surface reconstruction methods on ScanNet.

MethodChamfer ↓Prec ↑Recall ↑F-score ↑
20 views
VGGT + TSDF0.0770.7020.5340.605
IVGT + TSDF0.0760.7160.5690.632
VGGT + Poisson0.0530.6430.5920.615
IVGT + Poisson0.0520.6730.6300.651
IVGT (Ours)0.0470.6770.6800.678
40 views
VGGT + TSDF0.0550.6770.5750.621
IVGT + TSDF0.0520.6980.6070.648
VGGT + Poisson0.0480.6400.6430.640
IVGT + Poisson0.0480.6520.6650.658
IVGT (Ours)0.0460.7090.6820.693
60 views
VGGT + TSDF0.0490.6950.6130.651
IVGT + TSDF0.0500.7070.6340.668
VGGT + Poisson0.0460.6430.6790.659
IVGT + Poisson0.0450.6710.7080.689
IVGT (Ours)0.0420.7560.6940.723

We can observe that the conventional surface reconstruction methods generally benefit from more input views. This is because they aggregate depth or point-cloud observations across views. Additional views are more likely to reveal previously occluded regions, and overlapping observations can reduce the impact of errors in individual geometry predictions. Despite this, IVGT continues to achieve better reconstruction results across varying input numbers. The advantage of our implicit representation is particularly pronounced under sparse-view inputs, where IVGT recovers coherent and complete surfaces with substantially better reconstruction metrics. We believe these additional comparisons make our main experimental evaluation more complete and further demonstrate the advantages of our proposed method.

[1]Curless B, Levoy M. A volumetric method for building complex models from range images[C]//Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 1996: 303-312.

[2]Kazhdan M, Hoppe H. Screened poisson surface reconstruction[J]. ACM Transactions on Graphics (ToG), 2013, 32(3): 1-13.

W2 - Evaluation on the computational trade-offs of implicit reconstruction

Thanks for the suggestion. We systematically evaluate the computational trade-offs of our proposed method across varying numbers of input views. We report runtime details and memory consumption of IVGT on surface reconstruction and novel-view synthesis tasks, as shown in Table (b) and (c).

Table (b). Runtime details and memory consumption of IVGT on surface reconstruction task.

ViewsBackbone (s)Mesh query (s)Mesh extraction (s)Query numbersMemory (GiB) ↓
201.0922.5503.2591572608212.331
301.6873.4193.6941507530214.826
402.4404.2103.1511453767617.323
503.3624.9533.4481425698019.819
604.4425.8363.3691417397422.317

Table (c). Runtime details and memory consumption of IVGT on novel-view synthesis task.

ViewsBackbone (s)Render query (s)Query numbersMemory (GiB) ↓
201.0921.736649779218.987
301.6872.270649779222.132
402.4402.735649779225.267
503.3623.257649779228.409
604.4423.783649779231.551

In summary, as the number of input views increases, each query will aggregate features from more images, leading to higher runtime and memory consumption for downstream tasks. Nevertheless, both costs increase in a manageable manner with the input views, demonstrating the scalability of our method.

AuthorsOpenReview ↗

Author Response - Part 2

Q2 - Analysis on novel-view synthesis quality

Thanks for the question. To further analysis the observed perceptual quality degradation, we compare the rendered outputs of AnySplat and IVGT using HFEN [3] and DISTS [4], as shown in Table (d).

Table (d). Further quantitative analysis of the rendered outputs of AnySplat and IVGT.

MethodHFEN ↓DISTS ↓
AnySplat0.9370.130
IVGT (Ours)0.6660.241

As we have discussed in Appendix D, the geometric regularization we use to obtain smoother surfaces can, to some extent, hinder the recovery of fine geometric details. At the same time, the appearance modeling and rendering process can also affect perceptual quality. Gaussian-based representations have greater flexibility to fit image appearance without requiring all rendered details to be supported by a single coherent surface. In contrast, our SDF representation prioritizes geometric consistency across views, which supports coherent surface reconstruction and depth and normal estimation but may limit fine detail preservation. A similar trade-off between image quality and geometry is discussed in 2DGS [5] ("our regularization often involves a trade-off between image quality and geometry").

We are also exploring ways to improve the rendering quality while preserving surface consistency, thereby further enhance our method.

[3] Ravishankar, S., & Bresler, Y. (2010). MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE transactions on medical imaging, 30(5), 1028-1041.

[4] Ding, K., Ma, K., Wang, S., & Simoncelli, E. P. (2020). Image quality assessment: Unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence, 44(5), 2567-2581.

[5] Huang B, Yu Z, Chen A, et al. 2d gaussian splatting for geometrically accurate radiance fields[C]//ACM SIGGRAPH 2024 conference papers. 2024: 1-11.

Q3 - Origin of improvements over VGGT

Thanks for the suggestion. We finetune the original VGGT, and compare it with our proposed method using the same datasets and training budget.

Table (e). Pointmap reconstruction and Camera pose estimation.

DatasetMethodAcc mean ↓Acc median ↓Comp mean ↓Comp median ↓DatasetMethodATE ↓RPE_trans ↓RPE_rot ↓
NRGBDVGGT0.0200.0130.0160.007SintelVGGT0.1540.0630.466
NRGBDOurs0.0170.0120.0150.006SintelOurs0.1240.0660.512
7-ScenesVGGT0.0170.0090.0220.010ScanNetVGGT0.0320.0140.430
7-ScenesOurs0.0160.0070.0200.008ScanNetOurs0.0310.0140.407

We can observe that further training does improve the original VGGT. However, the implicit SDF representation also helps improve the shared geometric features. Therefore, the gains reported in Table 2 and 3 (in the paper) cannot be attributed solely to additional training data and supervision. It is worth noting that these gains are a positive side effect of our method. The primary contribution of IVGT is a queryable implicit representation that enables the extraction of complete mesh geometry and novel-view synthesis, using only unposed images as input without per-scene optizimation.

We hope the above response can help address your concerns. We have also made targeted revisions to the manuscript based on your suggestions. We are happy to answer any additional questions you may have.

Thanks for the response. The newly added experiments are useful for addressing my concerns. Still, I think this extension of geometry foundational model is an interesting idea, so I decided to raise my score to 3.

Dear Reviewer sQnG,

Thanks for your time and efforts. Your insightful review and valuable suggestions have significantly improved the clarity and quality of our work.

Best regards, Authors of Submission 12062

Reviewer_WM7rOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

IVGT combines a VGGT initialized backbone with an implicit SDF and appearance decoder. It reconstructs surfaces and renders novel views from unposed images without optimization at test time.

Critical strengths

Strong ScanNet mesh results at substantially lower inference time than per-scene optimization.

A unified representation supporting mesh extraction, rendering, and geometry estimation.

Critical weaknesses

A comparison with standard surface fusion using VGGT predictions is missing.

The training strategy lacks quantitative ablations, and the positional encoding experiments use a much shorter training schedule.

Geometry rendered from the implicit field is less accurate than directly decoded geometry on several benchmarks.

Questions for authors

How does IVGT compare with TSDF fusion using VGGT or IVGT's directly predicted depth and poses?

Can the authors quantify the effects of the second training stage and verify the encoding ablations with longer training?

3 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thanks for your thoughtful comments, questions and insightful suggestions. We provide detailed responses below.

W1 & Q1 - Comparison with standard surface fusion using VGGT predictions

Thanks for the suggestion. We use two standard methods, TSDF fusion [1] and Screened Poisson Surface Reconstruction [2], to obtain surfaces from points predicted by VGGT or IVGT. We then compared these surfaces quantitatively with those directly extracted by our method, as shown in Table (a).

Table (a). Comparison with standard surface fusion on ScanNet.

MethodChamfer ↓Prec ↑Recall ↑F-score ↑
20 views
VGGT + TSDF0.0770.7020.5340.605
IVGT + TSDF0.0760.7160.5690.632
VGGT + Poisson0.0530.6430.5920.615
IVGT + Poisson0.0520.6730.6300.651
IVGT (Ours)0.0470.6770.6800.678
40 views
VGGT + TSDF0.0550.6770.5750.621
IVGT + TSDF0.0520.6980.6070.648
VGGT + Poisson0.0480.6400.6430.640
IVGT + Poisson0.0480.6520.6650.658
IVGT (Ours)0.0460.7090.6820.693
60 views
VGGT + TSDF0.0490.6950.6130.651
IVGT + TSDF0.0500.7070.6340.668
VGGT + Poisson0.0460.6430.6790.659
IVGT + Poisson0.0450.6710.7080.689
IVGT (Ours)0.0420.7560.6940.723

We can observe that the conventional surface reconstruction methods generally benefit from more input views. This is because they aggregate depth or point-cloud observations across views. Additional views are more likely to reveal previously occluded regions, and overlapping observations can reduce the impact of errors in individual geometry predictions. Despite this, IVGT continues to achieve better reconstruction results across varying input numbers. The advantage of our implicit representation is particularly pronounced under sparse-view inputs, where IVGT recovers coherent and complete surfaces with substantially better reconstruction metrics. We believe these additional comparisons make our main experimental evaluation more complete and further demonstrate the advantages of our proposed method.

[1]Curless B, Levoy M. A volumetric method for building complex models from range images[C]//Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 1996: 303-312.

[2]Kazhdan M, Hoppe H. Screened poisson surface reconstruction[J]. ACM Transactions on Graphics (ToG), 2013, 32(3): 1-13.

AuthorsOpenReview ↗

Author Response - Part 2

W2 & Q2 - More ablation studies

Thanks for the suggestion. To further demonstrate the benefit of our second training stage and the effectiveness of our ray-depth embedding, we trained five variants on ScanNet and evaluated the quality of the surfaces extracted by these variants. As shown in Table (b), Stage One Only removes the second stage and instead extends the first-stage training to match our two-stage strategy. XYZ-PosEnc, XYZ-Embed, and Raydepth-Embed extend the training of the corresponding positional information injection strategies to a longer schedule, respectively. No Embed still fails to produce valid surfaces and is therefore omitted from the table.

Table (b). Positional Encoding & Training Strategy Ablation

MethodAcc ↓Comp ↓Chamfer ↓Prec ↑Recall ↑F-score ↑
Stage One Only (Full schedule)0.0580.0830.0710.5720.5120.540
XYZ-PosEnc0.0540.0570.0560.6270.6110.617
XYZ-Embed0.0520.0650.0580.6590.5870.620
Raydepth-Embed0.0460.0570.0520.6930.6130.651

We believe these ablations better quantify the effects of our second training stage and highlight the advantages of our ray-depth embedding.

W3 - Discussion on rendered geometry quality

Thanks for the question. As we have discussed in Limitations (in Appendix), the geometric regularization we use to obtain smoother surfaces can, to some extent, hinder the recovery of fine geometric details. Geometric consistency is essential for recovering a coherent 3D surface from multiple views, enabling reliable depth and normal estimation. However, enforcing this consistency usually smooth fine geometric structures, creating a practical trade-off between surface consistency and detail preservation, as also evidenced by 2DGS [3] ("our regularization often involves a trade-off between image quality and geometry").

Nevertheless, we value the benefits of implicit-field modeling, including the direct extraction of continuous, consistent surfaces and the ability to render from novel viewpoints beyond the input views. We are also exploring ways to improve the rendering quality and thereby further enhance our method.

[3] Huang B, Yu Z, Chen A, et al. 2d gaussian splatting for geometrically accurate radiance fields[C]//ACM SIGGRAPH 2024 conference papers. 2024: 1-11.

We hope the above response can help address your concerns. We have also made targeted revisions to the manuscript based on your suggestions. We are happy to answer any additional questions you may have.

Dear Reviewer WM7r,

Thank you for your thoughtful review and constructive suggestions, which have significantly improved the quality and clarity of our paper. We sincerely hope that our rebuttal has fully addressed all of your concerns. As the discussion period is ending soon, if you need any further clarification or additional information, please do not hesitate to contact us. We truly appreciate your time and effort devoted to reviewing our work.

Best regards, Authors of Submission 12062

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Author-Reviewer Discussion

Dear PCs, SACs, ACs and all of our reviewers,

We sincerely thank you for your time and effort in reviewing our paper and for the constructive feedback. We appreciate all of you for your comments highlighting the strengths of our work:

  • A well-motivated research problem and a clearly presented feed-forward framework for learning continuous implicit 3D scene representations from unposed multi-view images, which support downstream tasks such as surface extraction, pose estimation, novel-view synthesis, and depth and surface normal rendering without per-scene optimization.

  • Promising experimental results across multiple reconstruction tasks, particularly strong surface reconstruction performance on ScanNet that significantly outperforms other generalizable baselines and approaches the reconstruction quality of per-scene optimization methods, while achieving competitive performance in pointmap reconstruction, novel-view synthesis, depth estimation, and camera pose estimation.

We have provided the following additional experiments and analyses to address the reviewers’ concerns, and updated the manuscript accordingly:

  • Comparison with classical surface reconstruction. We compare IVGT with TSDF fusion and Screened Poisson Surface Reconstruction directly applied to VGGT-predicted geometry, demonstrating the benefit of learning an implicit representation against classical surface reconstruction methods (WM7r, sQnG).

  • Comparison with faster per-scene optimization baselines. We further compare IVGT with MonoSDF-grid, Neuralangelo, 2DGS, and QGS in reconstruction quality and optimization time to provide a more comprehensive evaluation against faster per-scene optimization methods (GepG).

  • More ablation on training strategy and positional encoding. We add ablations under the same extended training schedule to validate the benefits of the second training stage and our ray-depth embedding (WM7r).

  • Evaluation on the computational trade-offs. We provide a detailed runtime and memory analysis of our method on surface reconstruction and novel-view synthesis across varying numbers of input views, demonstrating the scalability of our method (sQnG).

  • Analysis of rendering quality. We use more metrics (HFEN and DISTS) to compare with AnySplat on novel-view synthesis, further discuss how geometric regularization, appearance modeling, and rendering affect the trade-off between surface consistency and fine-detail preservation (WM7r, sQnG).

  • Origin of improvements over VGGT. We compare IVGT with VGGT fine-tuned on the same datasets and training budget, showing that the improvements in pointmap reconstruction cannot be attributed solely to additional training data and supervision (sQnG).

  • Isolating the contribution of our proposed method. We reevaluate other generalizable surface reconstruction baselines using VGGT-predicted poses and geometry, showing that IVGT retains substantially better reconstruction quality and providing further evidence for the contribution of our proposed method (GepG).

  • Clarification of the novelty. We clarify that IVGT’s main novelty lies in its formulation of implicit visual geometry learning, which infers a shared and continuously queryable 3D field from unposed images and provides a common geometric basis for surface extraction, depth and normal rendering, and novel-view synthesis (GepG).

Thanks again to all of you for your time and efforts.

Best regards, Authors of Submission 12062

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.