- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes VGGT-Any, a framework for extending a frozen pinhole-based 3D foundation model to calibrated central cameras, including fisheye, panoramic, and heterogeneous camera systems.
The key idea is to decompose camera geometry into a canonical perspective-compatible component and a distortion component. The latter is encoded into a distortion token learned only from camera geometry, and the token conditions cross-attention adapters inserted into a frozen VGGT-Ω. The method therefore aims to preserve the original model’s pinhole reconstruction ability while adapting it to non-perspective cameras.
The paper also introduces LensZoo, a large synthetic dataset covering multiple central-camera families at shared physical poses. Experiments on fisheye, panoramic, heterogeneous-camera, and single-view settings show competitive or strong performance against camera-specific and camera-general baselines.
Critical strengths
-
The camera representation is a clear and meaningful contribution. Decomposing a central camera into a perspective-compatible canonical component and a distortion component provides a principled way to reuse a pretrained pinhole foundation model instead of retraining it for every camera type.
-
The empirical evaluation is broad. The paper evaluates fisheye, panoramic, heterogeneous-camera, and single-view settings, and compares against several closely related recent methods. LensZoo also provides useful controlled evaluation across camera models under shared poses.
Critical weaknesses
-
The “any central camera” claim appears somewhat stronger than the empirical evidence. The distortion-token pretraining includes all thirteen camera families considered by the method, with no camera family held out. The experiments therefore mainly demonstrate generalization to unseen parameters within known camera families, rather than to a genuinely unseen camera model family.
-
The comparison with some camera-general baselines is not fully information-matched. VGGT-Any receives the full known calibration/ray geometry, whereas Wid3R receives only a camera class. This is an important practical distinction, but it makes it difficult to attribute the performance gap solely to the proposed representation rather than to the stronger calibration information available to VGGT-Any.
Questions for authors
-
Can the authors clarify the intended meaning of “any central camera”? In particular, do you expect the distortion-token encoder to generalize to a camera family that was entirely absent from both geometry-token pretraining and adapter training?
-
Have the authors evaluated a leave-one-camera-family-out setting? Even a limited result on one held-out family would help distinguish interpolation within trained camera families from true camera-model generalization.
-
Could the authors clarify the fairness of the Wid3R comparison? Since VGGT-Any receives numerical calibration while Wid3R receives only a camera class, how much of the performance difference should be attributed to the proposed distortion representation versus the additional calibration information?
-
The adapters contain approximately 151M trainable parameters. Could the authors comment on whether similar performance can be retained with substantially smaller adapters, and how the adaptation cost compares with directly fine-tuning a camera-general model?
3 comments
3. Question 4: The adapters contain approximately 151M trainable parameters; can similar performance be retained with substantially smaller adapters, and how does adaptation cost compare with directly fine-tuning a camera-general model?
| Method | Trainable parameters (↓) | PointErr (↓) | Acc (↓) | Comp (↓) | AUC30 (↑) | RRA30 (↑) | RTA30 (↑) |
|---|---|---|---|---|---|---|---|
| VGGT-Omega (full fine-tuning) | 1,143.8M | 0.3125 | 0.1573 | 0.1073 | 0.4636 | 0.7486 | 0.7599 |
| Ours (VGGT-Any) | 151.2M | 0.1510 | 0.0811 | 0.0635 | 0.5949 | 0.8222 | 0.8670 |
- Ours vs. full fine-tuning: Our method updates
7.6xfewer trainable parameters than full fine-tuning of VGGT-Omega (151.2Mvs1,143.8M), freezes the pretrained backbone to preserve perspective accuracy, and adds+3.6%(+6.47 ms) inference latency on an 8-view window (+3.5%on 24 views). Both methods are evaluated under the same training steps on our devices across held-out LensZoo scenes from five distorted camera families (KB4, ERP, Smooth Angular, Equisolid, and EUCM). - Smaller adapters: Our 12-alternating-block design (
151.2Mparameters) balances multi-camera reconstruction accuracy against compute cost in our experiments. Smaller 4- or 8-block adapters cut compute cost, but do not match full pose and geometry accuracy across fisheye, panoramic, and heterogeneous cameras; placing adapters at all 24 blocks (302.3Mparameters) doubles adapter compute for only marginal accuracy gain. When targeting a smaller set of camera models, parameter count can be reduced further, though the main performance bottleneck remains the training data itself.
Thanks for the clarification. I will maintain my recommendation.
We are deeply grateful to the reviewer for the generous time, careful reading, and insightful comments devoted to evaluating our manuscript. Your constructive feedback has been invaluable in helping us strengthen our experimental analysis and improve the clarity of the paper. Below we address each weakness and question point by point.
1. Weakness 1, Question 1, and Question 2: The "any central camera" claim appears stronger than the empirical evidence since all thirteen camera families are included in pretraining; can the authors clarify its intended meaning and evaluate generalization to held-out camera families absent from training?
We thank the reviewer for pointing out where our writing on "any central camera" was not clear enough at the start of the paper. Although our training set sampled 13 parametric families, our formulation (Section 3.1) reconstructs any calibrated central camera in two parts: the canonical paraxial pinhole camera aligns non-perspective inputs with the frozen foundation model's perspective space, and the parameter-free angular distortion map captures lens-specific projection differences without family-specific formulas. For pinhole cameras the distortion term is zero, preserving exact perspective accuracy. To test generalization beyond those 13 families, we evaluated our submitted model without retraining on three central-camera families absent from both training stages (10 configurations each): Scaramuzza (Taylor omnidirectional with affine stretch), Mei-Rives (stereographic omnidirectional), and Fisheye624 (radial-tangential-thin-prism):
2. Weakness 2 and Question 3: VGGT-Any receives known calibration whereas Wid3R receives only a camera class; could the authors clarify the fairness of the Wid3R comparison and how much of the performance difference comes from the distortion representation versus additional calibration?
VGGT-Any takes camera calibration to build its canonical pinhole and distortion token, whereas Wid3R accepts only a camera-class label. When calibration is unavailable from sensor specifications, it can be estimated directly from the images and camera class. To compare both methods given the exact same inputs (RGB images and camera class, without ground-truth calibration), we ran VGGT-Any on 10 LensZoo settings using single-image intrinsics estimated by AnyCalib and GeoCalib:
With only RGB images and the camera class, VGGT-Any beats Wid3R on 7 of 10 settings, ties on Smooth Angular, and reduces 10-setting mean PointErr from
0.5641to0.1918(versus0.1063with ground-truth calibration). On settings such as Brown-Conrady where single-image estimation trails, the gap stems from single-image calibrator error: under controlled calibration perturbations on ScanNet++ fisheye scenes (α in {1%, 2%, 5%, 10%}, s=±1, f′=f(1+sα), c′=c+sαW, and k′=k(1+sα) or k+sαΔk over range Δk when ∣k∣<0.1Δk), VGGT-Any degrades gracefully: