VGGT-$\rho$: Accelerating Visual Geometry Inference via Adaptive View Selection

Feed-forward visual geometry models (e.g., VGGT-$\Omega$) have recently demonstrated impressive performance in scene reconstruction and camera pose estimation, but their inference cost grows rapidly with the number of views.We study whether scene reconstruction requires all candidate views to participate.Our study finds that sparse subsets preserve much of the available geometric coverage, while prioritizing feature-space coverage under a fixed distance threshold and restricting selection to local neighborhoods do not consistently improve reconstruction.These observations motivate VGGT-$\rho$, a training-free framework that selects views for high-resolution reconstruction using farthest-point sampling (FPS) over all candidate views in the feature space of low-resolution images. Image features are extracted from low-resolution images using the frozen VGGT-$\Omega$ image encoder; selection starts from a fixed reference view and repeatedly adds the candidate farthest from its nearest selected representative.The selected RGB images are passed in their relative input order to a frozen high-resolution reconstruction model.Across 7 Scenes, NRGBD, ScanNet-50, and Tanks and Temples, using only 15% of candidate views with VGGT-$\Omega$ increases mean Chamfer distance over the full-input scene domain by at most approximately 4 mm, while changing pose AUC@30 on the same selected views by $-0.1$ to $+1.5$ percentage points. On ScanNet-50, core-pipeline speedups reach $13.0\times$–$24.0\times$, with 63.3%–73.8% lower peak allocated GPU memory, including low-resolution encoding and selection overhead.Transfer experiments with FastVGGT and SparseVGGT further demonstrate the applicability of view selection across backbones, with quality trade-offs depending on the dataset and input budget. Project page: https://vggt-rho.github.io/.