acceptodds
Withdrawn from ICLR 2027

R0: Feed-Forward 3D Reconstruction for Arbitrary Central Cameras

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autonomous driving, robotics, and immersive VR/AR systems increasingly rely on fisheye and panoramic cameras for perception. However, existing feed-forward 3D reconstruction models are built on pinhole projection and degrade significantly on non-pinhole cameras. The root cause is that different cameras are treated as discrete categories, whereas all central cameras actually share the same ray space and differ only in how they unfold rays onto the image plane. We present R0, to our knowledge the first camera-agnostic feed-forward reconstruction model trained at scale across pinhole, fisheye, and panoramic cameras. By predicting a continuous ray field, constructing spherical neighborhood sampling, and conditioning cross-view attention on relative geometry, R0 unifies sampling, interaction, and output before projection. It natively handles different fields of view and unfolding functions, without requiring camera intrinsics, projection type, or undistortion at inference. Experiments on pinhole, fisheye, panoramic, and heterogeneous cameras show that R0 achieves state-of-the-art results on non-pinhole data while remaining competitive on pinhole data. Code, models, and large-scale data processing details will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Reconstruction, Arbitrary Cameras, Geometric Representation, Foundation Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_E2kWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes R0, a unified feed-forward 3D reconstruction model that can work with pinhole, fisheye, and panoramic cameras. It represents 3D points as viewing-ray directions + radial distances, and includes continuous ray-field prediction, spherical neighborhood sampling, and geometry-conditioned attention. Experiments show that R0 outperforms general-purpose baselines on non-pinhole reconstruction while retaining competitive performance on pinhole benchmarks.

Critical strengths
  • The targeted problem is real and well-formulated, where current feed-forward reconstruction models have limited generalization or formulation that can work with inputs from heterogeneous cameras.
  • The architectural design is coherent, where a unified ray space with a direction + radial distance formulation is consistently used.
  • The experiments are broad, evaluating reconstruction and pose/depth estimation across diverse datasets under diverse settings.
Critical weaknesses
  • On L242-243, "The geometric condition Gi\mathcal{G}_i of token ii contains its ray direction, 3D point, centered log radial distance, and validity indicator." However, there is no formulation that explains how Gi\mathcal{G}_i is obtained or how the ray direction, 3D point, and radial distance are predicted before the final output heads.
  • Camera-local directions and points require relative camera transformations to enable meaningful cross-view attention/comparisons. However, it seems unclear how R0 handles this part.
  • Currently, the main comparison is conducted with general-purpose, pinhole-oriented models (e.g., VGGT, MapAnything, Pi3) on non-pinhole images, which does not establish superiority over methods designed for that setting. More relevant methods like CAM3R and other fisheye and panoramic reconstruction methods should also be included in the comparison to sufficiently support the camera-agnostic property of R0. Currently, there seems to be no direct comparison with camera-specific methods included.
  • Particularly on Tab 3, a clear trade-off can be observed when enabling the joint processing of different camera types compared with Pi3. This makes the narrative about avoiding performance scarifice does not adequately acknowledge.
Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer E2kW

W1W2. Additional details of Geometry Attention

(1) Where does the initial geometry used by Geometry Attention come from? R0 uses two coarse-to-fine decoding stages. The first decoding uses only the original attention blocks and predicts initial rays, radial distances, and relative poses through the shared ray--radial and pose heads. The initial 3D points are obtained directly from the rays and radial distances; together, these predictions constitute the coarse geometry required by Geometry Attention. The second decoding reuses the same enhanced image features, introduces Geometry Attention in the later decoder blocks, and passes the result through the same output heads again to obtain the final rays, radial distances, poses, and confidence. We have clarified this consistently in Section 3.3 and Appendix A.1.

Within each view, Geometry Attention uses the initial rays, 3D points, centered radial distances, and valid support in that camera's coordinate frame, allowing image features with similar local geometry to interact. The valid support excludes the exterior of a circular fisheye image, padding, and other locations that do not correspond to physical imaging rays.

(2) How does cross-view interaction handle different camera coordinate frames? For cross-view interaction, the initial 3D points from all views are first transformed into the first-view reference frame using the relative poses predicted by the first decoding. They are then centered and scale-normalized before establishing cross-view geometric correspondences. Rays and radial distances are used primarily for within-view geometric interaction, whereas the transformed 3D points in the shared coordinate frame are used for cross-view interaction. Section 3.3 and Appendix A.1 now explicitly describe the two readings of the output heads and the geometry and coordinate frames used by the two types of Geometry Attention.

W3. Comparison with camera-specific methods We add direct comparisons with CAM3R, Wid3R, and several camera-specific monocular models such as Depth Any Panorama. (1) CAM3R uses a two-view feed-forward model and extends it to multiple views through an unreleased iterative backend. We therefore evaluate its released two-view model for pose estimation and point-map reconstruction on 7-Scenes, FIORD, and OB3D. (2) Wid3R requires the camera type in addition to the images. We use its corresponding built-in camera model for each dataset and compare multi-view reconstruction on OB3D, FIORD, and Deep360. (3) We also compare depth estimation on OB3D and WideDepth with camera-specific monocular models including UniK3D, DAC, DAP, and PanDA. All comparisons use the same test inputs and evaluation protocols.

(1) Two-view relative pose and point-map reconstruction

DatasetMethodRRA ↓RTA ↓mAA@30° ↑Acc. ↓Comp. ↓NC ↑
7-Scenes (Pinhole)Wid3R2.6877.79664.9840.4210.3280.778
CAM3R2.0728.11062.8510.2890.3180.703
R01.5684.47376.3150.2160.2150.810
FIORD (Fisheye)Wid3R2.0469.27555.2112.1333.0340.545
CAM3R9.47319.16635.9641.4873.5790.535
R01.3373.62976.6920.5262.7440.565
OB3D (Panorama)Wid3R0.3606.99352.2222.2911.0540.540
CAM3R3.72815.68043.3732.4113.7850.744
R00.9354.98980.7800.6250.4540.883

R0 achieves the best two-view reconstruction results on all three datasets and the best overall relative-pose performance.

(2) Multi-view reconstruction on wide-FoV cameras

DatasetMethodAcc. Mean ↓Acc. Median ↓Comp. Mean ↓Comp. Median ↓NC ↑
OB3DWid3R2.1981.5990.8360.5150.546
R00.4040.2340.2860.1810.839
FIORDWid3R0.5890.5110.5560.5040.723
R00.6950.6350.3930.3370.764
Deep360Wid3R2.5062.30012.9622.0020.525
R02.4731.0354.9071.1330.819

We also add comparisons with camera-specific monocular models including UniK3D, DAC, DAP, and PanDA.

(3) Depth estimation comparison with camera-specific monocular models

MethodRGB onlyOB3D AbsRel ↓OB3D RMSE ↓OB3D δ1 ↑
UniK3DYes0.1163.2540.892
DAC-IndoorNo0.1801.6470.736
DAC-OutdoorNo0.3352.1570.484
DAPYes0.0882.5770.950
PanDAYes0.5205.3650.337
R0Yes0.0751.0850.943
The results have been added to Sections 4.3.1 and 4.3.3, with complete evaluation protocols in Appendix A.3.2--A.3.3.

W4. Claim of avoiding performance degradation (1) By avoiding performance degradation, we mean that R0 does not show a substantial decline on pinhole benchmarks relative to several advanced general-purpose pinhole models. R0 replaces Pi3's point-map head with a ray--radial head, changing the output from three-channel (X,Y,Z)(X,Y,Z) to a four-channel representation. Its weights therefore do not exactly match Pi3 even at initialization, and R0 should not be expected to outperform Pi3 on every pinhole dataset. (2) The original large-scale training experienced several server instabilities and job interruptions. We reran the complete end-to-end training and reevaluated the new checkpoint. The updated results show that R0 reaches state of the art on several pinhole datasets and is not substantially worse than other models on the remaining datasets. The updated pinhole reconstruction, pose-estimation, and depth results are listed below.

(1) Pinhole point-map reconstruction

Method7-Scenes Sparse Acc. ↓Comp. ↓NC ↑7-Scenes Dense Acc. ↓Comp. ↓NC ↑
VGGT0.0430.0600.7660.0190.0270.693
Pi30.0450.0710.7470.0130.0160.694
MapAnything0.0980.1620.7070.0610.0480.658
Depth Anything 30.0560.0690.7370.0210.0230.679
Pi3X0.0530.0810.7510.0130.0170.697
VGGT-Ω\Omega0.0400.0520.7460.0290.0300.673
R00.0440.0710.7310.0150.0120.676

(2) Pinhole camera-pose estimation

MethodSintel ATE ↓RPE-T ↓RPE-R ↓
Pi30.0740.0390.279
Pi3X0.1250.0410.295
VGGT0.1720.0620.471
VGGT-Ω\Omega0.1020.0330.222
MapAnything0.2260.0770.642
Depth Anything 30.1140.0560.469
R00.0920.0490.506

(3) Pinhole depth estimation

(3a) Monocular depth estimation (per-image scale alignment)

MethodStructured3D-Pinhole AbsRel ↓RMSE ↓δ1 ↑
VGGT0.1550.2780.889
Pi30.4801.0170.895
MapAnything0.3560.8670.880
Depth Anything 30.1770.3070.882
VGGT-Ω\Omega0.1700.3040.883
Pi3X0.2370.5050.901
R00.1280.2810.912

(3b) Video depth estimation (sequence-shared scale and shift)

MethodBonn AbsRel ↓RMSE ↓δ1 ↑ScanNet AbsRel ↓δ1 ↑Sintel AbsRel ↓RMSE ↓δ1 ↑
VGGT0.0490.2490.9740.0380.9840.2265.5650.683
Pi30.0440.2210.9750.0260.9900.2103.9780.728
Pi3X0.0380.2110.9750.0260.9910.1884.0270.737
MapAnything0.0500.2390.9750.0530.9720.2904.5850.688
Depth Anything 30.0490.2230.9730.0530.9660.2665.4900.654
VGGT-Ω\Omega0.1150.3490.9320.0360.9800.3565.7400.625
R00.0410.2060.9760.0250.9920.2173.7040.685

These results cover pinhole reconstruction, pose estimation, monocular depth, and video depth. R0 achieves the best completeness on 7-Scenes Dense, the second-best ATE for Sintel pose estimation, and the best AbsRel and δ1\delta_1 on Structured3D-Pinhole. It also obtains the best RMSE and δ1\delta_1 on Bonn, the best AbsRel and δ1\delta_1 on ScanNet, and the best RMSE for Sintel video depth. Thus, after extending to fisheye and panoramic cameras, R0 remains comparable to or better than advanced pinhole models across multiple pinhole tasks, rather than avoiding degradation on only a single pinhole benchmark. The quantitative results appear in Sections 4.1.1--4.1.3. We also add qualitative visualizations with ERP, pinhole, and fisheye inputs in “Additional reconstruction comparisons” in Appendix A.5, covering different scenes and numbers of input views.

5. Reproducibility

We will release the complete large-scale training code, evaluation code, and model weights to support reproducibility and future research.

Dear Reviewer E2kW,

As the discussion phase is nearing its end, we kindly invite you to review our rebuttal. We have clarified the source of the initial geometry and the cross-view coordinate transformations, added comparisons with camera-specific methods, and revised our discussion of the pinhole-performance trade-off. We would appreciate your feedback and welcome any further questions.

Reviewer_qhUrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

R0 extends a Pi3-based feed-forward multi-view reconstruction model to pinhole, fisheye, panoramic, and mixed-camera inputs. It represents 3D points using per-pixel ray directions and radial distances, predicts a continuous ray field, and uses ray-space neighborhood sampling and geometry-conditioned attention for within-view and cross-view interaction. The model is trained on 31 data sources and requires only RGB images at inference, without camera intrinsics or projection labels. Experiments cover point-cloud reconstruction, camera pose, and depth estimation. Gains are particularly strong on several non-pinhole benchmarks, whereas results on standard pinhole benchmarks are mixed.

Critical strengths
  1. A unified design that does not require calibration at inference. A single model processes RGB inputs with different projections without requiring camera intrinsics or a projection label. The ray-direction/radial-distance representation also provides an explicit geometric output.
  2. Large gains on several non-pinhole tasks. For example, FIORD pose AUC@30 increases from the best listed baseline's 44.401 to 81.638, and OB3D point-cloud reconstruction Acc. improves from 2.166 to 0.670, that's a big improvement.
Critical weaknesses
  1. The closest camera-aware methods are missing from quantitative comparisons. CAM3R has studied uncalibrated ray and radial-distance prediction, while Wid3R addresses wide-FoV multi-view reconstruction. Although R0's large-scale cross-projection training and specific modules may still be distinct contributions, comparisons mainly against pinhole-oriented general models do not establish the magnitude of its advance over prior cross-camera methods. Matched-input and matched-protocol evaluations are needed.
  2. The pinhole-retention and component-attribution claims are not fully supported by controlled evidence. In Table 6, full R0 has worse ScanNet-Pinhole Acc. than pretrained Pi3 (0.063 versus 0.053); in Table 3, its Sintel ATE is 0.175 versus Pi3's 0.074. Appendix A.4 also reports different batch sizes or stopping points for Pi3-FT, the head-only variant, and the remaining ablations. Claims of preserving pinhole performance and of gains attributable to individual components therefore need more qualification and matched-budget controls.
Questions for authors
  1. Can the authors compare R0 with CAM3R and Wid3R under matched camera inputs, evaluation protocols, and, where feasible, training data? Which claimed contributions remain distinct after those comparisons?
  2. Can the authors provide ablations matched for training budget, batch size, and stopping criteria, and quantify the cross-projection versus pinhole-performance trade-off on multiple pinhole benchmarks?
  3. At inference, how does Geometry Attention obtain its 3D points and radial distances before the final output head has produced them? How does Ray Canonical Sampling implement differentiable inverse projection when a predicted ray field may not have a unique inverse?
3 comments

Q1W1. Comparison with CAM3R and Wid3R

We add direct comparisons with CAM3R and Wid3R on zero-shot datasets. (1) CAM3R consists of a two-view feed-forward model and a multi-view optimization backend. Because its multi-view version has not been released, we compare two-view pose estimation and reconstruction. R0 achieves the best two-view reconstruction results on all three datasets and the best overall relative-pose performance. (2) In addition to images, Wid3R requires the camera type as input. For each dataset, we evaluate Wid3R using its corresponding built-in camera model and separately compare multi-view reconstruction with Wid3R on panoramic and fisheye datasets.

(1) Two-view pose estimation and point-map reconstruction

DatasetMethodRRA ↓RTA ↓mAA@30° ↑Acc. ↓Comp. ↓NC ↑
7-Scenes(Pinhole)Wid3R2.6877.79664.9840.4210.3280.778
CAM3R2.0728.11062.8510.2890.3180.703
R01.5684.47376.3150.2160.2150.810
FIORD (Fisheye)Wid3R2.0469.27555.2112.1333.0340.545
CAM3R9.47319.16635.9641.4873.5790.535
R01.3373.62976.6920.5262.7440.565
OB3D(Panorama)Wid3R0.3606.99352.2222.2911.0540.540
CAM3R3.72815.68043.3732.4113.7850.744
R00.9354.98980.7800.6250.4540.883

(2) Multi-view reconstruction comparison with Wid3R on wide-FoV cameras

DatasetMethodAcc. Mean ↓Acc. Median ↓Comp. Mean ↓Comp. Median ↓NC ↑
OB3DWid3R2.1981.5990.8360.5150.546
R00.4040.2340.2860.1810.839
FIORDWid3R0.5890.5110.5560.5040.723
R00.6950.6350.3930.3370.764
Deep360Wid3R2.5062.30012.9622.0020.525
R02.4731.0354.9071.1330.819

In addition, Section 4.3.3 now includes comparisons with camera-specific monocular models, including UniK3D, DAC, DAP, and PanDA. These comparisons further clarify how R0 differs from existing methods: R0 takes RGB images alone and supports single-view, two-view, and joint multi-view inference within one feed-forward model. Our core contributions remain large-scale cross-camera training and a unified ray-space reconstruction system spanning sampling, interaction, and output. We also add qualitative visualizations under ERP, pinhole, and fisheye inputs in “Additional reconstruction comparisons” in Appendix A.5, covering different scenes and numbers of input views.

Q2W2. Training-budget-matched ablations

Firstly, we clarify that every ablation model in the original paper was trained until its validation performance converged; the different step counts resulted from different convergence rates. To further eliminate this factor, we retrained and reevaluated the ablations under matched conditions. The training setup is as follows: (1) every model is trained for 20,000 optimizer steps without early stopping, while both the best and last checkpoints are saved to ensure sufficient convergence; (2) every run uses four RTX 4090 GPUs and the same global batch size of 24; and (3) each ablation is evaluated using its best checkpoint.

MethodHolo360D-ERPHolo220-DualFisheyeOmniScene-HeterogeneousScanNet-Pinhole
Acc. ↓Comp. ↓NC ↑CD ↓Acc. ↓Comp. ↓NC ↑CD ↓Acc. ↓Comp. ↓NC ↑CD ↓Acc. ↓Comp. ↓NC ↑CD ↓
Pi33.1321.5960.5434.7280.4331.1830.6041.6160.5032.5140.5393.0180.0320.0340.9430.066
Pi3-FT0.3390.5540.6460.8930.2510.1450.6830.3960.3760.3340.8170.7100.0670.0560.9120.123
R0 w/o Samp. & Attn.0.1530.1430.9030.2960.1210.1090.9220.2300.2670.3850.8680.6510.0450.0460.9010.091
R0 w/o Samp.0.1150.1120.9280.2280.0930.0870.9450.1800.2300.3100.8910.5400.0350.0340.9330.070
R0 w/o Attn.0.1500.1380.9110.2870.1170.1010.9240.2180.2490.3650.8740.6140.0420.0410.9090.083
Full R00.1080.1080.9330.2160.0870.0830.9490.1700.2400.2910.8950.5310.0330.0340.9330.067

Q3. Geometry Attention and Ray Canonical Sampling

How does Geometry Attention obtain geometry before the final output? R0 uses two coarse-to-fine decoding stages. The first decoding uses Pi3's original attention blocks to decode the features into initial rays, radial distances, and relative poses, with the initial 3D points obtained as X=ρr\mathbf X=\rho\mathbf r. For cross-view interaction, the camera-local 3D points are transformed into a common reference frame using the relative poses predicted by the first decoding. The second decoding reuses the same features and applies Geometry Attention in the later decoder blocks using this complete initial geometry, producing the refined output. Geometry Attention therefore uses the coarse geometry produced by the first decoding rather than the as-yet-unavailable final output. We have clarified this consistently in Section 3.3 and Appendix A.1.

How does Ray Canonical Sampling implement differentiable inverse projection? We do not assume that the predicted ray field has an analytic inverse. Instead, for each canonical ray, we directly solve for its corresponding position in the original image. The process has two steps. We first find the position with the closest direction on a low-resolution ray grid and use it as a coarse initialization. We then bilinearly interpolate the ray field around that position and iteratively adjust the image coordinates so that the interpolated ray direction aligns with the target ray. The resulting continuous coordinates are used to bilinearly sample the original image, allowing gradients to propagate from the sampled image back to the predicted ray field.

If multiple positions have similar ray directions, we select the local solution with the smallest angular error. The predicted valid imaging support excludes invalid locations such as the exterior of a circular fisheye image and padding. For ERP images, the horizontal dimension is treated periodically so that locations on opposite sides of the seam are not incorrectly considered nonadjacent. Only the selection of the local region is discrete; once the region is fixed, coordinate refinement and image sampling are both continuous and differentiable. We have added this procedure to “Ray Canonical Sampling” in Appendix A.1.

4. Open Source Release

We will release the complete large-scale training code, evaluation code, and model weights to support reproducibility and future research.

Dear Reviewer qhUr, As the discussion phase is nearing its end, we kindly invite you to review our rebuttal. We have added comparisons with CAM3R and Wid3R, clarified the ablation protocols, and explained the implementations of Geometry Attention and Ray Canonical Sampling. We would be happy to discuss any remaining questions or provide further clarification.

Reviewer_CmwGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for feed-forward 3D reconstruction from unposed, uncalibrated image sets, where the images may have come from different types of central cameras. That is, one image might be an equirectangular panorama, another might be a fisheye image, and a third might be a pinhole camera image. The approach is to adapt Pi3 by replacing its 3-channel point map head with a 4-channel ray-radius head, alongside the standard Pi3 confidence map and camera pose outputs. This description from the paper contradicts the rest of the paper, which states that the new head actual predicts a much larger (unspecified) number of channels, which are interpreted as spherical harmonic bases, coefficients and domain, tangent-plane residuals and validity mask, as well as the radial distance. The problem that this paper addresses is a significant one, since known the restriction to a pinhole camera model is somewhat limiting for applications. The claimed contributions are (a) the ray-space representation; (b) The continuous ray field parameterized by spherical harmonics; and (c) strong generalization across cameras. The approach is trained on 31 datasets, including new fisheye projection augmentations, for 5 days on 64 GPUs, and is evaluated on 4 reconstruction datasets, 6 camera pose estimation datasets, and 5 video/monocular depth estimation datasets, demonstrating good performance across a range of tasks and cameras.

Critical strengths
  1. Motivation. The argument that a unified representation is required so that images from arbitrary camera models can be used by the same foundation model is a very strong one. However, it is not clear why a point map is not that unifying representation, since it does not depend on the camera model. The related argument that existing models make some hidden assumptions about the mapping from pixels to rays is also one that I wholeheartedly agree with. It would be great to see this analysed systematically: this paper appears poised to do so. I also find the argument that having mixed sensors as input to the same network is important to be compelling, for the reason given in the paper that robots typically have multiple camera types on the one platform, which really ought to share information. This is a fantastic motivation.

  2. Experiments. The results (with some caveats) appear strong, particularly the pose estimation results for non-pinhole camera datasets. The results for non-pinhole datasets are where the gains really appear, compared to other generalist methods.

  3. Approach. The approach appears to be coherent and sensible: predicting per-pixel ray directions and radial distances should (and indeed does) work to address this problem.

Critical weaknesses
  1. Claims. The evidence provided for the two main claims is somewhat weak.

    1. The central claim of the paper is that training feed-forward 3D reconstruction models, such as Pi^3 or VGGT, with image data from mixed camera types leads to worse performance on each data type than if they had been fine-tuned separately: "mixed training degrades the model’s original performance on pinhole images, yet still falls short of camera-specific models separately fine-tuned on fisheye or panoramic data" [L061-063]. This motivates the paper's methodological alterations to Pi^3. However, the evidence for this claim is limited: Table 6 shows results on Pi^3 fine-tuned on a mixed 3-dataset subset of the 31 training datasets, which does worse than the pretrained model on pinhole data (ScanNet-pinhole) but much better on non-pinhole data. So we have some limited evidence for the first half of the claim, but no evidence for the second half ("still falls short of camera-specific models separately fine-tuned on fisheye or panoramic data"). Moreover, there are many reasons why fine-tuning with a different data mixture changes the pretrained results: training on a new subset of pinhole data may also degrade performance on ScanNet-pinhole merely due to overfitting to a different distribution.
    2. Regarding this same claim, not only is the empirical evidence lacking, but no theoretical backing is given to this claim either. The argument made by the authors is as follows. "Current models reason primarily in the image plane, where image coordinates and visual patterns are coupled with projection-specific ray geometry. The same image-plane evidence can therefore carry incompatible geometric meanings under different camera models. Learning a new pixel-to-ray relationship interferes with the perspective prior already encoded by the network, creating a trade-off between retaining existing capabilities and acquiring new ones" [L065]. However, no evidence is given to demonstrate that these models (especially Pi^3) actually do reason in the image plane. Pi^3 is a transformer-based architecture, which has no pixel-neighbourhood inductive bias, so it is unclear where this bias would be coming from. Moreover, the very purpose of the output representation (the pointmap) proposed in DuSt3R is to avoid assuming a camera model, since the 3D point predictions do not need to lie on pinhole camera rays. There are certainly some reasons I can think of where image-plane reasoning come in (such as in the convolutional DINO encoder), but this part is left unchanged by the authors. Missing this analysis is a missed opportunity for the paper, and would provide some solid evidence for the central claim.
    3. The abstract claims that this is the "first camera-agnostic feed-forward reconstruction model trained at scale across pinhole, fisheye and panoramic cameras" [L031]. However, UniK3D (Piccinelli et al., CVPR 2025) is a prior camera-agnostic feed-foward 3D reconstruction model trained at scale (26 training datasets) across pinhole, fisheye and panoramic cameras. The claim requires finessing, such as by specifying "multi-view reconstruction model".
  2. Contributions. The technical contributions specified in the paper have somewhat limited novelty, having appeared in existing literature. Looking at them in order:

    1. The ray-space representation. The ray + radial distance representation appears in existing work, in particular UniK3D (Piccinelli et al., CVPR 2025) that predicts rays and radial distances in the monocular setting for the same reason as in this paper, MapAnything (Keetha et al., 2025) that predicts ray directions, ray depths and poses in the multi-view setting where calibration is an optional input not a requirement, and DAv3 (Lin et al., 2025) that predicts ray maps and depth. It is an over-claim to suggest that this representation is a novel contribution.
    2. The continuous ray field parameterized by spherical harmonics. This idea appears to come directly from UniK3D (Piccinelli et al., 2025), which also uses the DINO CLS token to predict global intrinsic properties using spherical harmonics. The one difference I can see is that the proposed method also predicts a per-patch ray residual in tangent space, to model non-global phenomena. The similarities and differences from this approach are not made clear in the paper.
  3. Evaluation.

    1. To assess the claim "[Pi^3 trained on mixed data] still falls short of camera-specific models separately fine-tuned on fisheye or panoramic data", the Pi3-FT model needs to be compared to camera-specific models separately fine-tuned on fisheye or panoramic data.
    2. To demonstrate the contribution over and above the approach proposed in UniK3D, a quantitative comparison with UniK3D's approach applied to the Pi3 backbone is warranted.
    3. A baseline that predicts the camera type from the images and routes to different specialist models (MoE) would be ideal.
    4. An ablation demonstrating the importance of the per-pixel ray residual is required, since this is what is added on top of UniK3D.
    5. Table 6 indicates that almost all of the performance improvement is attributable to fine-tuning on the mixed dataset.
  4. Clarity. The paper does not present its approach or results very clearly, making it challenging to understand precisely. Some examples follow.

    1. The paper implies that a very small change is made to Pi^3 ("We implement this representation on Pi3 by replacing its point map head with a four-channel ray–radial head that outputs rays and distances"), but the next section immediately contradicts this, saying that the new head actually predicts a much larger (unspecified) number of channels, which are interpreted as spherical harmonic bases, coefficients and domain, tangent-plane residuals and validity mask, as well as the radial distance. This is quite confusing.
    2. In Fig. 2, the program flow is very unclear. It is a critical diagram for communicating the approach. More than any other reader, your reviewers want to understand exactly what you've done so that you get the credit for your work and ideas. This relates to the missing details comments below: it's challenging to give a fair assessment without understanding the full picture.
    3. The meaning of "the valid ray domain" [L232] is never specified.
    4. It is not clear in the tables which columns refer to pinhole-only data vs non-pinhole data vs mixed data. For the latter, it is not clear what the proportion of the different types is. While this information can be reverse-engineered from the datasets, it would help the reader if it was made clear in the tables themselves.
  5. Missing methodological details. The method is significantly under-specified, especially in the main paper, but even when considering the appendix as well, which I read in detail in an effort to better understand the approach. I would appreciate a full mathematical specification, at least in the appendix, for the approach; for example, the mathematical definitions of a, Y(.) and omega(.) that appear to be the main predicands.

    1. The method section abruptly says that we have the "3D point, centered log radial distance" [L243], but up to this point in the method, only the ray directions have been estimated. No mention has been made of the radial distance or 3D point, so where do they come from?
    2. The CLS token is used to predict "field of view, spherical center, and continuous parameters controlling the radial curve r(θ). These parameters define a differentiable mapping from pixel coordinates u to spherical directions ωv(u)". None of these elements are specified, so the reader has no way to understand how this works.
    3. "resulting features are injected into the native patch tokens through a learned gate whose strength combines a geometric-reliability prior with a learned residual" [L235]. Here again there are no more details provided: how does this actually work?
    4. The "kernels" of Eq. 4 are undefined mathematically, but are needed to understand the contribution. In particular, one kernel is stated to be the relative distance between 3D points, but this value does not saturate, and so would dominate the content term. So I assume the kernel is somewhat different to what has been stated?
    5. No positional encoding is specified: if there is none, this should be mentioned.
  6. Missing training details.

    1. Section 3.4 states that the loss is composed of 5 sub-losses (3 undefined) and training is composed of two stages. However, Section A.2 reports 5 additional undefined losses, and 4 stages of training in total. This is inconsistent, and makes it very challenging to understand the approach, let alone reproduce it. Everything should be clearly defined, if only in the appendix with appropriate forward references in the main paper.
    2. Moreover, why is this 4-stage training needed? Does joint training collapse? If so, why?
    3. Similarly, why are ground-truth labels required as model inputs during training? Could they not be better used to supervise intermediate predictions early in training?
  7. Missing evaluation details.

    1. The reconstruction metrics (Acc, Comp, NC) are undefined in the paper and appendix. It's particularly odd that the paper indicates that lower accuracy and lower completeness are better (Table 1). Moreover, accuracy usually requires some threshold, which has not been defined here. This point is quite serious, since it undermines any credence that the reader might have in the results of Table 1.
Questions for authors
  1. What is your empirical evidence for the central claim that "mixed training degrades the model’s original performance on pinhole images, yet still falls short of camera-specific models separately fine-tuned on fisheye or panoramic data", and what theoretical reason do you give for this?

  2. How do you distinguish your ray+radial-distance representation from the existing literature?

  3. How do you distinguish your ray field approach from UniK3D?

  4. (See questions in the missing details sections above)

  5. In Table 6, why is the Accuracy number for R0 for Holo360D-ERP 0.145, but is 2.7 in Tab 1, when it is trained on much more data in Tab 1 and would therefore expect to be more accurate?

4 comments

Q1W1. Central claim regarding mixed-camera training

This claim contains two parts.

(1) Does mixed-camera training reduce the original pinhole performance? Our controlled ablation provides direct evidence. Under the same ablation data and evaluation protocol, the original Pi3 obtains an Acc. of 0.053 on ScanNet-Pinhole, whereas mixed-camera fine-tuned Pi3 obtains 0.215. Meanwhile, Pi3-FT improves Acc. on Holo360D-ERP and Holo360D-Fisheye from 2.435 and 0.426 to 0.348 and 0.252, respectively. The controlled experiment therefore directly supports the following conclusion: in this setting, direct mixed-camera fine-tuning substantially improves non-pinhole inputs but introduces a clear retention--adaptation trade-off.

(2) Does mixed-camera training remain inferior to separately fine-tuned camera-specific models? The original paper did not train ERP-only and fisheye-only Pi3 baselines strictly matched to Pi3-FT, so this statement lacked direct empirical evidence. We accept the reviewer's point and remove this unverified comparison from Section 1. The revised claim is limited to what the controlled experiment directly supports: mixed-camera fine-tuning substantially improves non-pinhole performance but reduces the original pinhole performance, and it remains clearly weaker than the complete R0 trained on the same mixed-camera data. The new comparisons with Wid3R, CAM3R, and camera-specific monocular models in Sections 4.3.1 and 4.3.3 evaluate R0 against existing methods; we do not use them as evidence for the above Pi3 counterfactual.

Our explanation of the “pinhole prior” is as follows. Although a point map can serve as a camera-model-independent output, the model's input features are still organized in the image plane, where the same pixel displacement corresponds to different angular displacements under pinhole, fisheye, and panoramic projections. In addition, perspective statistics learned from large amounts of pinhole data, as well as the zz-depth outputs used by some methods, can introduce priors associated with pinhole projection.

We also thank the reviewer for pointing out the issue with our positioning. In Section 1, “Introduction,” we revise the positioning to “the first feed-forward multi-view reconstruction model for arbitrary central cameras.”

Q2W2. Difference from prior ray--radial representations Our method is not merely a change from Pi3's output to a ray--radial output; it is a complete reconstruction system in which rays are used throughout sampling, geometric interaction, and output. (1) During encoding, Ray Canonical Sampling samples local image views, passes them through the encoder to obtain sampled features, and gives the model comparable FoV receptive fields across different images. The sampled and native features are then fused into enhanced encoding features. (2) Decoding follows a coarse-to-fine strategy. The first decoding maps the enhanced features to initial rays, radial distances, and relative poses. The second decoding reuses the same enhanced features and uses the complete initial geometry through Geometry Attention to guide cross-view interaction, producing a refined reconstruction. (3) Because Ray Canonical Sampling requires rays as input, we construct a Ray Field Predictor that predicts a per-pixel ray field and valid imaging support for each image, avoiding reliance on auxiliary inputs.

Q3W2. Difference between the Ray Field Predictor and UniK3D

(1) The predicted rays serve different purposes. UniK3D is a monocular 3D prediction method whose rays and depth constitute the model's final output. In R0, the Ray Field Predictor is an independent intermediate component. Its predicted rays are first used by Ray Canonical Sampling to construct unified local observations for images under different projections. R0's final refined rays and radial distances are predicted separately by the ray--radial head after the second decoding. The Ray Field Predictor and ray--radial output head are therefore two distinct components: the former supplies the camera geometry required during reconstruction, whereas the latter produces the final reconstruction.

(2) UniK3D predicts rays, whereas R0 also predicts valid imaging support. Pixels in an input image tensor do not necessarily correspond to physical camera rays. Examples include black regions outside a circular fisheye image, invalid regions created when a nonrectangular projection is stored in a rectangular tensor, and padding introduced by aspect-ratio-preserving resizing. If these pixels participate in ray matching and geometric reasoning alongside valid pixels, Ray Canonical Sampling may sample from invalid regions, and subsequent geometric interaction may treat those regions as valid 3D observations. The Predictor therefore also outputs per-pixel valid imaging support. This support constrains canonical chart centers and ray-to-pixel correspondences and remains a validity constraint for the initial geometry and second decoding, ensuring that only physical imaging regions participate in sampling and geometric interaction.

(3) The ray fields are modeled differently. We acknowledge and now state explicitly in Section 3.2 and Appendix A.1 that the spherical-harmonic ray representation in R0's global branch is inspired by UniK3D, which we cite at the corresponding locations. Building on this idea, R0 treats the global representation as a smooth ray-field anchor and adds a spatial branch conditioned on patch features and global context. This branch predicts per-pixel tangent-plane residuals that further correct the global rays on the unit sphere, modeling higher-order lens distortion and local deviations that cannot be described by a single set of global camera parameters. The complete Predictor therefore consists of a global SH anchor, per-pixel tangent residuals, and a valid-support head, rather than only a set of global camera parameters.

(4) Additional Ray Field Predictor ablation. In addition to the existing experiment in “Continuous-FoV ray prediction” in Appendix A.5, we add a comparison between the full Predictor and a version without tangent residuals (the UniK3D-style variant) in the same subsection. Both versions use the same data, training budget, and test protocol. The table below reports the mean ray angular error at six representative FoVs.

Predictor60° ↓90° ↓120° ↓180° ↓240° ↓360° ↓
w/o Tangent Residual0.1790.2750.2590.7290.8252.294
Full Predictor0.1760.2730.2460.5750.7011.127

All values are mean angular errors in degrees.The two methods are comparable from 60∘60^\circ to 120∘120^\circ, while tangent residuals provide clearer benefits at large FoVs, notably reducing the error on 360∘360^\circ ERP from 2.294∘2.294^\circ to 1.127∘1.127^\circ. This result shows that local tangent-plane residuals complement the global SH anchor by modeling local ray deviations that are difficult to represent globally.

In addition, Appendix A.5, “Continuous-FoV ray prediction,” already contains the corresponding experimental figure. Under the same training protocol and across five random seeds, it compares (1) our continuous Ray Field Predictor, (2) Fixed Classification, which identifies a discrete projection class and uses the corresponding class prototype, and (3) Direct MLP, a parameter-matched baseline that directly regresses per-pixel rays. The original experiment shows that the classifier performs well only near its discrete prototypes, while Direct MLP struggles to fit a structured spherical ray field. Our Predictor maintains lower and more stable angular errors across the continuous FoV range. This existing experiment and the tangent-residual ablation above validate the roles of continuous structured representation and local residual refinement, respectively.

Q4W4W5W6. Method, training, and evaluation details

We thank the reviewer for identifying implementation and experimental details that were insufficiently specified in the original manuscript. We have systematically added the following information to Sections 3.2--3.4, Appendix A.1 “Architecture Details,” Appendix A.2 “Training Details,” and Appendix A.3 “Evaluation Protocols.”

(1) Method and supervision. Sections 3.2--3.4 and Appendix A.1 now provide the complete data flow of the Ray Field Predictor, Ray Canonical Sampling, and Geometry Attention, and explicitly distinguish the intermediate rays/valid support produced by the Predictor from the final ray--radial reconstruction produced by the second decoding. Appendix A.2.1, “Supervision,” defines the ray, radial-distance, point, pose, confidence, and normal losses, including their weights and valid-pixel domains. All per-pixel losses are computed only where the ground-truth geometry is valid and lies within the physical imaging support, excluding sky, padding, and invalid depth. Appendix A.2.1 also explains that the reconstruction and Predictor paths use gradient-decoupled objectives: they share frozen encoder features but do not propagate loss gradients into each other.

(2) Training procedure and optimization. Section 3.4 and Appendix A.2.3, “Training Settings,” now specify the Stage A-to-B training procedure. In Stage A, the reconstruction path first performs 10,000 one-decoder updates, after which the second decoding and Geometry Attention are enabled for up to 15,000 updates. During the same stage, the Predictor independently learns the SH anchor, tangent residual, and valid-support branches under its own objective. Stage B freezes the Predictor, replaces the teacher condition with its predicted rays/support, and trains the complete two-decoder reconstruction model for another 15,000 updates. The long image side is resized to 518 pixels; the short side preserves the original aspect ratio and is aligned to the encoder patch size. Each group is sampled from a single scene, while the number of views follows the dataset-specific protocol. All reconstruction stages use AdamW with (β1,β2)=(0.9,0.95)(\beta_1,\beta_2)=(0.9,0.95), weight decay 0.05 for matrix parameters and no decay for biases or one-dimensional parameters, together with linear warm-up, cosine decay, and global gradient clipping at 1.0.

(3) Evaluation protocols. Appendix A.3.1--A.3.3 now includes a taxonomy of comparison methods, unified metric definitions, and dataset-specific evaluation settings.

Q4W7. Definition of reconstruction metrics

Acc. is the mean distance from each predicted point to its nearest ground-truth point and measures the accuracy of the predicted geometry. Comp. is the mean distance from each ground-truth point to its nearest predicted point and measures how completely the reconstruction covers the ground-truth surface. Acc. therefore does not require an additional distance threshold. We have made these definitions explicit in Appendix A.3.2, “Evaluation Settings--Point Map Reconstruction.”

Q5. Data scope of the ablation and the different Holo360D results in Table 6

Table 6 is a controlled ablation study conducted on partial data rather than the complete datasets used by the main experiments. Specifically, we select subsets of scenes from Holo360D, ScanNet, and OmniScene and construct separate train, validation, and test splits within these selected scenes. Thus, both the training data and the test data in Table 6 differ from those used in Table 1: the Holo360D result in Table 6 is evaluated on our ablation test split, whereas Table 1 follows the main-paper evaluation protocol and uses the formal model trained on all 31 sources. The two Holo360D numbers therefore do not evaluate the same model or the same test set and should not be compared directly. We have added the selected scenes, custom train/validation/test splits, and corresponding evaluation settings to Appendix A.4, “Ablation Details.”

Additional experiments

We further add direct comparisons with CAM3R, Wid3R, and several camera-specific monocular models. All methods use the same test inputs and evaluation protocols; R0 uses only RGB images in every experiment.

(1) Two-view comparison with CAM3R and Wid3R. CAM3R releases a two-view feed-forward model, while its multi-view backend has not been open-sourced. We therefore compare two-view relative pose and point-map reconstruction on 7-Scenes, FIORD, and OB3D.

DatasetMethodRRA ↓RTA ↓mAA@30° ↑Acc. ↓Comp. ↓NC ↑
7-Scenes (Pinhole)Wid3R2.6877.79664.9840.4210.3280.778
CAM3R2.0728.11062.8510.2890.3180.703
R01.5684.47376.3150.2160.2150.810
FIORD (Fisheye)Wid3R2.0469.27555.2112.1333.0340.545
CAM3R9.47319.16635.9641.4873.5790.535
R01.3373.62976.6920.5262.7440.565
OB3D (Panorama)Wid3R0.3606.99352.2222.2911.0540.540
CAM3R3.72815.68043.3732.4113.7850.744
R00.9354.98980.7800.6250.4540.883

R0 achieves the best two-view reconstruction results on all three datasets and the best overall relative-pose performance.

(2) Wide-FoV multi-view reconstruction comparison with Wid3R. Wid3R requires the camera type in addition to RGB images; we evaluate it using the corresponding built-in camera model for each dataset.

DatasetMethodAcc. Mean ↓Acc. Median ↓Comp. Mean ↓Comp. Median ↓NC ↑
OB3DWid3R2.1981.5990.8360.5150.546
R00.4040.2340.2860.1810.839
FIORDWid3R0.5890.5110.5560.5040.723
R00.6950.6350.3930.3370.764
Deep360Wid3R2.5062.30012.9622.0020.525
R02.4731.0354.9071.1330.819

(3) Depth comparison with camera-specific monocular models. We also compare with UniK3D, DAC, DAP, and PanDA on OB3D, using a single ERP image as input for every method.

MethodRGB onlyOB3D AbsRel ↓OB3D RMSE ↓OB3D δ1 ↑
UniK3DYes0.1163.2540.892
DAC-IndoorNo0.1801.6470.736
DAC-OutdoorNo0.3352.1570.484
DAPYes0.0882.5770.950
PanDAYes0.5205.3650.337
R0Yes0.0751.0850.943

These additional experiments show that R0 is not only strong relative to general-purpose pinhole models, but also remains competitive under matched protocols against the two-view CAM3R model, Wid3R with camera-type inputs, and several camera-specific monocular models. The results have been added to Sections 4.3.1 and 4.3.3, with complete evaluation protocols in Appendix A.3.1--A.3.3. We also add qualitative visualizations with ERP, pinhole, and fisheye inputs in “Additional reconstruction comparisons” in Appendix A.5, covering different scenes and numbers of input views.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.