acceptodds
Withdrawn from ICLR 2027

VGGT-: Accelerating Visual Geometry Inference via Adaptive View Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feed-forward visual geometry models (e.g., VGGT-) have recently demonstrated impressive performance in scene reconstruction and camera pose estimation, but their inference cost grows rapidly with the number of views.We study whether scene reconstruction requires all candidate views to participate.Our study finds that sparse subsets preserve much of the available geometric coverage, while prioritizing feature-space coverage under a fixed distance threshold and restricting selection to local neighborhoods do not consistently improve reconstruction.These observations motivate VGGT-, a training-free framework that selects views for high-resolution reconstruction using farthest-point sampling (FPS) over all candidate views in the feature space of low-resolution images. Image features are extracted from low-resolution images using the frozen VGGT- image encoder; selection starts from a fixed reference view and repeatedly adds the candidate farthest from its nearest selected representative.The selected RGB images are passed in their relative input order to a frozen high-resolution reconstruction model.Across 7 Scenes, NRGBD, ScanNet-50, and Tanks and Temples, using only 15% of candidate views with VGGT- increases mean Chamfer distance over the full-input scene domain by at most approximately 4 mm, while changing pose AUC@30 on the same selected views by to percentage points. On ScanNet-50, core-pipeline speedups reach –, with 63.3%–73.8% lower peak allocated GPU memory, including low-resolution encoding and selection overhead.Transfer experiments with FastVGGT and SparseVGGT further demonstrate the applicability of view selection across backbones, with quality trade-offs depending on the dataset and input budget. Project page: https://vggt-rho.github.io/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D reconstruction, Visual Geometry Foundation Models, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_o8ToOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training-free acceleration of feed-forward 3D reconstruction models (VGGT family) by giving them fewer input views. For each candidate image, it computes its low-resolution DINO features then uses farthest-point sampling (FPS) in this feature space to pick a fixed fraction of views (15%), starting from the first frame. The frozen VGGT model runs only on the selected views. On 7 Scenes, NRGBD, ScanNet-50 and Tanks and Temples, using 15% of the views keeps reconstruction and pose quality close to the full input. On ScanNet-50 it gives 13–24× speedups and 63–74% lower peak memory, including the selection cost.

Critical strengths
  1. Simple, training-free and plug-and-play. The selector works without any training, and reduces runtime by 13–24× and peak memory by 63–74% on ScanNet. The reported times include the selection cost, which makes the efficiency numbers honest. This matters because reconstruction from long image sets is a real bottleneck for these models.
Critical weaknesses
  1. Uniform sampling gives almost the same results as the proposed FPS on DINO features, which weakens the main contribution. In the paper's own ablation (Tab. 5, 6), Uniform is roughly the same as the proposed sampling strategy, while also being faster. The main tables (Tab. 1–4) only compare Full vs. + Ours and do not include Uniform at all. The speed and memory gains come from using fewer views, which any subsampling gives. So the paper does not show that its selection rule, the main contribution, really matters.

  2. All test sets are densely captured, mostly indoor videos, where uniform sampling already works well. Except for Tanks and Temples, all datasets (7 Scenes, NRGBD, ScanNet-50) are dense indoor videos with very redundant frames. For a small room like those in NRGBD or ScanNet-50, there is no need to reconstruct the geometry from 500 or 1000 views. In my experience, the more overlapping views are fed to VGGT, the larger the chance that the model outputs layered artifacts in the geometry, which makes the metrics worse. This explains why selecting far fewer views gives better quality here, both in numbers and visually. So the lesson here is that these datasets just need far fewer views. It does not explain why FPS on DINO features is needed. A proper demonstration would run the method on challenging, very long, large-scale inputs (e.g., the sequences used in VGGT-Long or TTT3R) and show that the proposed selection beats both the baselines and simple uniform sampling.

  3. The method solves a simpler task, so the speedup comparison is not fair. The method only predicts depth and poses for the selected views, and pose is evaluated only on these views. As noted above, for small rooms, feeding fewer views even makes it easier for the model to get good results. In contrast, Full inference and acceleration methods like FastVGGT and SparseVGGT give outputs for all N frames. Many downstream tasks need every frame results. So the speedup compares two different outputs, and I do not think this comparison is fair.

Questions for authors
  1. Could author add Uniform, Random and fixed-stride rows to Tables 1–4?

  2. Could the method work well on very long, large-scale inputs (e.g., the sequences used in VGGT-Long or TTT3R) and other settings where uniform sampling should fail (non-uniform camera speed, unordered image collections). Does FPS beat Uniform there?

  3. All three backbones are VGGT-based: original VGGT, FastVGGT and SparseVGGT. So the claim that view selection applies "across backbones" (abstract) is tested on VGGT model family only. Why not other feed-forward models with global cross-view attention, such as Depth Anything 3 or Pi3?

Reviewer_tj7pOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks whether feed-forward visual-geometry models need to process every available image at high resolution. VGGT-ρ\rho first encodes all candidate images at low resolution using the frozen DINO-initialized image encoder of VGGT-Ω\Omega. Each image is represented by a spatially pooled descriptor, and reference-initialized farthest-point sampling (FPS) selects a fixed fraction of views that are diverse in descriptor space. Only the selected images enter the expensive high-resolution multi-view model, in their original relative order. The method is training-free and is evaluated with VGGT-Ω\Omega, FastVGGT, and SparseVGGT on 7 Scenes, NRGBD, ScanNet-50, and Tanks and Temples. At 15% retention, it often preserves reconstruction and pose quality while reducing VGGT-Ω\Omega core-pipeline latency by 13.0×13.0\times–24.0×24.0\times and peak allocated GPU memory by 63.3%–73.8%.

Critical strengths
  1. Clear representation. The paper is easy to follow and read.
Critical weaknesses
  1. The validation of generalization ability is limited. The candidate datasets are largely dense sequences with substantial temporal redundancy. For sparse settings such as autonomous driving, or settings with considerable appearance changes such as unordered Internet/photo collections, looped trajectories, and dynamic scenes, the generalization ability of the method requires more detailed validation. Besides, the user-specified retained-view fraction ρ depends on view coverage and sparsity, and its generalization to different input settings is also a concern.
  2. Problematic task formulation. The method predicts depth and pose only for 15% of the frames and evaluates mainly scene coverage. This substantially simplifies the original problem: standard multi-view reconstruction often requires per-frame poses and depths, and producing these outputs for all frames is precisely why cross-view attention becomes expensive. It makes the contribution and significance of the solution rather limited.
Questions for authors

See weakness.

Reviewer_amtkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VGGT-rho, an adaptive frame selection method for 3D reconstruction. The motivation is that using a properly sampled subset of images, the reconsrtuction model can reconstruct the scenes with performance on par with full set of the images. This sampling strategy can accerate the inference (with fewer images), reduce the resource request, and achieve comparable performance. The authors start from some naive baselines and propose their feature based FPS method. Experiments on several benchmarks indicate the effectiveness of the idea.

Critical strengths
  • This is a well-motivated paper. The motivation is clear. Given tons of frames, it might be unnecessary to reconstruct the scene with all images. How to select subsets from the images so that the model can achieve comparable reconstruction quality is an insteresting research topic from my point of view.
  • The experiments show that some promising results in this direction. Using a subset of images can somehow achieve comparable results with using all images.
Critical weaknesses
  • The proposed DINO based method is not obviously better than the simple uniform sampling. Its performance is very close to the uniform sampling and it's a bit slower due to the DINO forward as shown in Table 5. It seems that it's not necessary to adopt the proposed solution and a simple uniform sampling can almost solve this problem.
  • For the proposed method, all those submodules are kind of standard, the low-resolution descriptor, pooling, cos similarity, and FPS. The issue is not that those standard submodules cannot contribute to a good method, but it should be somehow reasonable to be applied. For a common video, I don't see the necessarity. But it would be interesting to see that this method can work better than the unified sampling startegy in some specific scenario, for example, a camera that is not moving, a camera that is moving with changing speed, or a camera revisiting the same place.
  • Though using selected frame can reconstruct the full scene, we can never estimate the camera pose for those unsampled frames. However, I think it's a drawback in the setting level of this task, but still if the paper can try to deal with this issue, it can be a good point.
Questions for authors

Please address the concerns in the Critical Weaknesses section, particularly the practical advantage over uniform sampling and the implications of producing predictions only for selected frames. These are the main considerations that could affect my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.