acceptodds
Under review as a conference paper at ICLR 2027

Extending 3D Foundation Models from to Any Central Camera with Distortion Awareness Token

Abstract

3D foundation models, such as Visual Geometry Grounded Transformer (VGGT), have shown superior performance in inferring 3D attributes of a scene from several views. However, they do not generalize to non-perspective cameras, limiting their usage in other camera models, such as fisheye and panoramic cameras. To this end, we propose VGGT-Any, which extends VGGT to any type of central camera and to heterogeneous camera systems without retraining the base model. At the core of VGGT-Any is a novel camera representation: it decomposes camera geometry into a canonical space, which follows the usual perspective-camera formulation, and a distortion space, which expresses each camera's specific projection. We further introduce a distortion token, which is injected into the frozen foundation model to reconstruct scenes from various central cameras, or even from heterogeneous camera systems. Moreover, since existing datasets each cover only a limited range of camera types, we construct LensZoo, a synthetic indoor dataset in which any central camera model can be synthesized at shared poses, enabling controlled comparisons across camera models. Our evaluations show that VGGT-Any is competitive with camera-specific and fine-tuned baselines on real fisheye and panoramic benchmarks, and exactly retains the base model's pinhole reconstruction ability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D foundation models, feed-forward 3D reconstruction, camera models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nTzvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VGGT-Any, a framework for extending a frozen pinhole-based 3D foundation model to calibrated central cameras, including fisheye, panoramic, and heterogeneous camera systems.

The key idea is to decompose camera geometry into a canonical perspective-compatible component and a distortion component. The latter is encoded into a distortion token learned only from camera geometry, and the token conditions cross-attention adapters inserted into a frozen VGGT-Ω. The method therefore aims to preserve the original model’s pinhole reconstruction ability while adapting it to non-perspective cameras.

The paper also introduces LensZoo, a large synthetic dataset covering multiple central-camera families at shared physical poses. Experiments on fisheye, panoramic, heterogeneous-camera, and single-view settings show competitive or strong performance against camera-specific and camera-general baselines.

Critical strengths
  1. The camera representation is a clear and meaningful contribution. Decomposing a central camera into a perspective-compatible canonical component and a distortion component provides a principled way to reuse a pretrained pinhole foundation model instead of retraining it for every camera type.

  2. The empirical evaluation is broad. The paper evaluates fisheye, panoramic, heterogeneous-camera, and single-view settings, and compares against several closely related recent methods. LensZoo also provides useful controlled evaluation across camera models under shared poses.

Critical weaknesses
  1. The “any central camera” claim appears somewhat stronger than the empirical evidence. The distortion-token pretraining includes all thirteen camera families considered by the method, with no camera family held out. The experiments therefore mainly demonstrate generalization to unseen parameters within known camera families, rather than to a genuinely unseen camera model family.

  2. The comparison with some camera-general baselines is not fully information-matched. VGGT-Any receives the full known calibration/ray geometry, whereas Wid3R receives only a camera class. This is an important practical distinction, but it makes it difficult to attribute the performance gap solely to the proposed representation rather than to the stronger calibration information available to VGGT-Any.

Questions for authors
  1. Can the authors clarify the intended meaning of “any central camera”? In particular, do you expect the distortion-token encoder to generalize to a camera family that was entirely absent from both geometry-token pretraining and adapter training?

  2. Have the authors evaluated a leave-one-camera-family-out setting? Even a limited result on one held-out family would help distinguish interpolation within trained camera families from true camera-model generalization.

  3. Could the authors clarify the fairness of the Wid3R comparison? Since VGGT-Any receives numerical calibration while Wid3R receives only a camera class, how much of the performance difference should be attributed to the proposed distortion representation versus the additional calibration information?

  4. The adapters contain approximately 151M trainable parameters. Could the authors comment on whether similar performance can be retained with substantially smaller adapters, and how the adaptation cost compares with directly fine-tuning a camera-general model?

3 comments

We are deeply grateful to the reviewer for the generous time, careful reading, and insightful comments devoted to evaluating our manuscript. Your constructive feedback has been invaluable in helping us strengthen our experimental analysis and improve the clarity of the paper. Below we address each weakness and question point by point.

1. Weakness 1, Question 1, and Question 2: The "any central camera" claim appears stronger than the empirical evidence since all thirteen camera families are included in pretraining; can the authors clarify its intended meaning and evaluate generalization to held-out camera families absent from training?

We thank the reviewer for pointing out where our writing on "any central camera" was not clear enough at the start of the paper. Although our training set sampled 13 parametric families, our formulation (Section 3.1) reconstructs any calibrated central camera in two parts: the canonical paraxial pinhole camera aligns non-perspective inputs with the frozen foundation model's perspective space, and the parameter-free angular distortion map captures lens-specific projection differences without family-specific formulas. For pinhole cameras the distortion term is zero, preserving exact perspective accuracy. To test generalization beyond those 13 families, we evaluated our submitted model without retraining on three central-camera families absent from both training stages (10 configurations each): Scaramuzza (Taylor omnidirectional with affine stretch), Mei-Rives (stereographic omnidirectional), and Fisheye624 (radial-tangential-thin-prism):

Unseen camera familyVGGT-Any PointErr (↓)VGGT-Omega PointErr (↓)Wid3R PointErr (↓)
Scaramuzza0.1860.2810.371
Mei-Rives0.1260.3360.516
Fisheye6240.1930.2260.271

2. Weakness 2 and Question 3: VGGT-Any receives known calibration whereas Wid3R receives only a camera class; could the authors clarify the fairness of the Wid3R comparison and how much of the performance difference comes from the distortion representation versus additional calibration?

VGGT-Any takes camera calibration to build its canonical pinhole and distortion token, whereas Wid3R accepts only a camera-class label. When calibration is unavailable from sensor specifications, it can be estimated directly from the images and camera class. To compare both methods given the exact same inputs (RGB images and camera class, without ground-truth calibration), we ran VGGT-Any on 10 LensZoo settings using single-image intrinsics estimated by AnyCalib and GeoCalib:

SettingWid3R (class + image) PointErr (↓)VGGT-Any (estimated cal from class + image) PointErr (↓)VGGT-Any (true cal) PointErr (↓)
KB40.40360.19740.1002
ERP (parameter-free)1.07820.10590.1059
UCM0.54880.11890.0665
EUCM0.95150.13240.0934
Division0.40740.29990.0814
Double Sphere0.75640.14980.1374
Heterogeneous rig0.95740.24330.1630
Smooth Angular0.20400.20400.0970
FOV0.11660.13710.1102
Brown-Conrady0.21730.32980.1081

With only RGB images and the camera class, VGGT-Any beats Wid3R on 7 of 10 settings, ties on Smooth Angular, and reduces 10-setting mean PointErr from 0.5641 to 0.1918 (versus 0.1063 with ground-truth calibration). On settings such as Brown-Conrady where single-image estimation trails, the gap stems from single-image calibrator error: under controlled calibration perturbations on ScanNet++ fisheye scenes (α\alpha in {1%, 2%, 5%, 10%}, s=±1s = \pm 1, f′=f(1+sα)f' = f(1 + s\alpha), c′=c+sαWc' = c + s\alpha W, and k′=k(1+sα)k' = k(1 + s\alpha) or k+sαΔkk + s\alpha\Delta k over range Δk\Delta k when ∣k∣<0.1Δk|k| < 0.1\Delta k), VGGT-Any degrades gracefully:

Perturbed groupGT (PointErr ↓ / AUC30 ↑)1% (PointErr ↓ / AUC30 ↑)2% (PointErr ↓ / AUC30 ↑)5% (PointErr ↓ / AUC30 ↑)10% (PointErr ↓ / AUC30 ↑)
Focal0.0660 / 0.93030.0643 / 0.92970.0703 / 0.92740.0722 / 0.92440.0753 / 0.9208
Distortion0.0660 / 0.93030.0642 / 0.92390.0634 / 0.91870.0731 / 0.89920.1060 / 0.8040
Principal point0.0660 / 0.93030.0734 / 0.90740.0711 / 0.90260.0970 / 0.87700.1796 / 0.7816
All parameters0.0660 / 0.93030.0639 / 0.91530.0713 / 0.90220.1038 / 0.84530.1497 / 0.7565

3. Question 4: The adapters contain approximately 151M trainable parameters; can similar performance be retained with substantially smaller adapters, and how does adaptation cost compare with directly fine-tuning a camera-general model?

MethodTrainable parameters (↓)PointErr (↓)Acc (↓)Comp (↓)AUC30 (↑)RRA30 (↑)RTA30 (↑)
VGGT-Omega (full fine-tuning)1,143.8M0.31250.15730.10730.46360.74860.7599
Ours (VGGT-Any)151.2M0.15100.08110.06350.59490.82220.8670
  1. Ours vs. full fine-tuning: Our method updates 7.6x fewer trainable parameters than full fine-tuning of VGGT-Omega (151.2M vs 1,143.8M), freezes the pretrained backbone to preserve perspective accuracy, and adds +3.6% (+6.47 ms) inference latency on an 8-view window (+3.5% on 24 views). Both methods are evaluated under the same training steps on our devices across held-out LensZoo scenes from five distorted camera families (KB4, ERP, Smooth Angular, Equisolid, and EUCM).
  2. Smaller adapters: Our 12-alternating-block design (151.2M parameters) balances multi-camera reconstruction accuracy against compute cost in our experiments. Smaller 4- or 8-block adapters cut compute cost, but do not match full pose and geometry accuracy across fisheye, panoramic, and heterogeneous cameras; placing adapters at all 24 blocks (302.3M parameters) doubles adapter compute for only marginal accuracy gain. When targeting a smaller set of camera models, parameter count can be reduced further, though the main performance bottleneck remains the training data itself.

Thanks for the clarification. I will maintain my recommendation.

Reviewer_m3wFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VGGT-Any for calibrated multi-view reconstruction across central camera models. It decomposes camera geometry into a canonical pinhole model and a distortion map. A geometry VAE compresses this information into a token. The token conditions adapters in a frozen VGGT model. The paper also introduces LensZoo for controlled evaluation across camera models.

Decoupling reconstruction models from camera hardware is highly valuable in industry. Deploying models such as VGGT across different devices often requires sensor-specific adaptation. This approach could reduce the need to fine-tune the base model for each sensor.

Critical strengths
  1. The performance looks good.
  2. The formulations of the different calibrations are valuable.
Critical weaknesses
  1. The main contribution is not sufficiently isolated. Table 5 replaces the token with random noise or bypasses the adapters at inference. Neither condition is retrained. These results show that the trained model relies on its conditioning. They do not establish that the proposed decomposition and VAE are necessary.
  2. Generalization to unseen camera models remains unclear.
Questions for authors

What benefit comes from the canonical–distortion decomposition itself?

2 comments

We are deeply grateful to the reviewer for the generous time, careful reading, and insightful comments devoted to evaluating our manuscript. Your constructive feedback has been invaluable in helping us strengthen our experimental analysis and improve the clarity of the paper. Below we address each weakness and question point by point.

1. Weakness 1 and Question 1: The main contribution is not sufficiently isolated because Table 5 does not retrain, which does not establish that the proposed decomposition and VAE are necessary; what benefit comes from the canonical-distortion decomposition itself?

  1. Benefit of the canonical-distortion decomposition:
    • VGGT-Omega's original camera token jointly reconstructs camera pose and perspective intrinsics KiK_i. Without canonical-distortion decomposition, supporting non-perspective cameras requires retraining KiK_i prediction in VGGT-Omega, which degrades base-model performance and impairs VGGT-Omega's ability to reconstruct camera intrinsics.
    • Frozen VGGT-Omega predicts a perspective camera K^i\hat K_i and planar depth Di(p)D_i(p), then converts planar depth to ray range via ri(p)=Di(p)∥K^i−1p~∥2r_i(p) = D_i(p)\lVert\hat K_i^{-1}\tilde p\rVert_2 before unprojecting along calibrated rays Φi⋆(p)\Phi_i^\star(p) (Eq. 6). A raw non-pinhole ray map Φ⋆(p)\Phi^\star(p) defines no perspective focal length. Our decomposition Φ⋆=D∘Φref\Phi^\star = D \circ \Phi_{\mathrm{ref}} extracts each camera's first-order paraxial pinhole KrefK_{\mathrm{ref}} (canonical space) alongside the higher-order angular warp DD (distortion space, zero for pinholes so perspective accuracy is unchanged at 0.0672 LensZoo PointErr). Without KrefK_{\mathrm{ref}}, retraining on raw rays can supervise pose (Ri,ti)(R_i, t_i) and range, but gives VGGT-Omega's camera head no perspective field-of-view target. To test this with retraining, we trained two identical ray-conditioned adapter models on frozen VGGT-Omega (LensZoo Pinhole, KB4, and ERP, both unprojecting along calibrated rays): one supervising the camera head with our paraxial KrefK_{\mathrm{ref}} field of view, and one omitting KrefK_{\mathrm{ref}} supervision. Without KrefK_{\mathrm{ref}} anchoring, K^i\hat K_i drifts by 38% to 56% from the paraxial scale, misscales Di(p)∥K^i−1p~∥2D_i(p)\lVert\hat K_i^{-1}\tilde p\rVert_2, and worsens both 3D geometry and pose across trained and held-out camera families:
Camera splitWith canonical KrefK_{\mathrm{ref}} (PointErr ↓ / AUC30 ↑)Without canonical KrefK_{\mathrm{ref}} (PointErr ↓ / AUC30 ↑)
Trained camera families0.1143 / 0.76780.1239 / 0.7578
Held-out camera families0.1306 / 0.75520.1374 / 0.7501
  1. Why the geometry-only VAE is needed:
    • Stage 1 trains the VAE on closed-form camera geometry alone without RGB images, so the distortion token cannot entangle camera intrinsics with scene appearance.
    • Variational regularization maps diverse lens warps into a continuous latent space centered at the zero-distortion pinhole origin, supporting zero-shot interpolation to unseen camera families.
    • Instead of feeding the uncompacted 4 x 512 x 512 geometry map (1,048,576 floats, or 1,024 patch tokens per frame) into cross-attention at every adapter block, the VAE compacts each camera into a single 1,024-float vector (4 KiB, mapped to 8 conditioning tokens), which caches offline per sensor and adds +3.6% (+6.47 ms) latency on an 8-view window.

2. Weakness 2: Generalization to unseen camera models remains unclear.

We evaluated our submitted model without retraining on three camera families unseen in both training stages (10 configurations each): Scaramuzza (Taylor omnidirectional with affine stretch), Mei-Rives (stereographic omnidirectional), and Fisheye624 (radial-tangential-thin-prism):

Unseen camera familyVGGT-Any PointErr (↓)VGGT-Omega PointErr (↓)Wid3R PointErr (↓)
Scaramuzza0.1860.2810.371
Mei-Rives0.1260.3360.516
Fisheye6240.1930.2260.271

Thanks for the clarification. I will maintain my recommendation.

Reviewer_EweSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a framework that extends pretrained 3D foundation models to different central camera models, including fisheye and panoramic cameras, without modifying the backbone. The key idea is to decompose camera geometry into two components: (1) a canonical space aligned with the perspective camera model, and (2) a distortion space that captures camera-specific projection differences. The authors introduce a geometry-only VAE to encode camera distortion into a compact token, which is injected into the frozen VGGT-Omega through cross-attention adapters. Additionally, the authors construct a large-scale synthetic dataset covering various camera models.

Critical strengths
  • The paper addresses an important practical challenge: extending 3D foundation models to non-perspective cameras while preserving their original pinhole reconstruction capabilities.
  • The proposed camera decomposition is intuitive and geometrically motivated. Learning distortion tokens independently of image content is an interesting design choice.
Critical weaknesses
  • Limited generalization validation
    • Although the paper claims to support arbitrary central cameras, all thirteen camera families are included during training. Generalization to unseen camera families remains unclear.
  • The method relies on known camera calibration.
    • Both the distortion token and the final 3D reconstruction depend on accurate camera calibration, but its robustness to calibration errors is not evaluated.
Questions for authors
  • Could the authors provide experiments on unseen camera families to demonstrate generalization?
  • Could the authors provide experiments with perturbed camera parameters to evaluate robustness to calibration errors?
1 comment

We are deeply grateful to the reviewer for the generous time, careful reading, and insightful comments devoted to evaluating our manuscript. Your constructive feedback has been invaluable in helping us strengthen our experimental analysis and improve the clarity of the paper. Below we address each weakness and question point by point.

1. Weakness 1 and Question 1: All thirteen camera families are included during training and generalization to unseen camera families remains unclear; could the authors provide experiments on unseen camera families to demonstrate generalization?

We evaluated our submitted model without retraining on three camera families unseen in both training stages (10 configurations each): Scaramuzza (Taylor omnidirectional with affine stretch), Mei-Rives (stereographic omnidirectional), and Fisheye624 (radial-tangential-thin-prism):

Unseen camera familyVGGT-Any PointErr (↓)VGGT-Omega PointErr (↓)Wid3R PointErr (↓)
Scaramuzza0.1860.2810.371
Mei-Rives0.1260.3360.516
Fisheye6240.1930.2260.271

2. Weakness 2 and Question 2: Both the distortion token and the final 3D reconstruction depend on accurate camera calibration; could the authors provide experiments with perturbed camera parameters to evaluate robustness to calibration errors?

On ScanNet++ fisheye scenes, for relative error α\alpha in {1%, 2%, 5%, 10%} and sign s=±1s = \pm 1, we perturb focal length f′=f(1+sα)f' = f(1 + s\alpha), principal point c′=c+sαWc' = c + s\alpha W (for image width WW), and distortion coefficients k′=k(1+sα)k' = k(1 + s\alpha), using k′=k+sαΔkk' = k + s\alpha\Delta k over parameter range Δk\Delta k when ∣k∣<0.1Δk|k| < 0.1\Delta k. Both distortion-token encoding and ray unprojection receive the perturbed calibration, and predictions are scored against ground truth:

Perturbed groupGT (PointErr ↓ / AUC30 ↑)1% (PointErr ↓ / AUC30 ↑)2% (PointErr ↓ / AUC30 ↑)5% (PointErr ↓ / AUC30 ↑)10% (PointErr ↓ / AUC30 ↑)
Focal0.0660 / 0.93030.0643 / 0.92970.0703 / 0.92740.0722 / 0.92440.0753 / 0.9208
Distortion0.0660 / 0.93030.0642 / 0.92390.0634 / 0.91870.0731 / 0.89920.1060 / 0.8040
Principal point0.0660 / 0.93030.0734 / 0.90740.0711 / 0.90260.0970 / 0.87700.1796 / 0.7816
All parameters0.0660 / 0.93030.0639 / 0.91530.0713 / 0.90220.1038 / 0.84530.1497 / 0.7565

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.