- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes R0, a unified feed-forward 3D reconstruction model that can work with pinhole, fisheye, and panoramic cameras. It represents 3D points as viewing-ray directions + radial distances, and includes continuous ray-field prediction, spherical neighborhood sampling, and geometry-conditioned attention. Experiments show that R0 outperforms general-purpose baselines on non-pinhole reconstruction while retaining competitive performance on pinhole benchmarks.
Critical strengths
- The targeted problem is real and well-formulated, where current feed-forward reconstruction models have limited generalization or formulation that can work with inputs from heterogeneous cameras.
- The architectural design is coherent, where a unified ray space with a direction + radial distance formulation is consistently used.
- The experiments are broad, evaluating reconstruction and pose/depth estimation across diverse datasets under diverse settings.
Critical weaknesses
- On L242-243, "The geometric condition of token contains its ray direction, 3D point, centered log radial distance, and validity indicator." However, there is no formulation that explains how is obtained or how the ray direction, 3D point, and radial distance are predicted before the final output heads.
- Camera-local directions and points require relative camera transformations to enable meaningful cross-view attention/comparisons. However, it seems unclear how R0 handles this part.
- Currently, the main comparison is conducted with general-purpose, pinhole-oriented models (e.g., VGGT, MapAnything, Pi3) on non-pinhole images, which does not establish superiority over methods designed for that setting. More relevant methods like CAM3R and other fisheye and panoramic reconstruction methods should also be included in the comparison to sufficiently support the camera-agnostic property of R0. Currently, there seems to be no direct comparison with camera-specific methods included.
- Particularly on Tab 3, a clear trade-off can be observed when enabling the joint processing of different camera types compared with Pi3. This makes the narrative about avoiding performance scarifice does not adequately acknowledge.
Questions for authors
See weakness.
3 comments
W4. Claim of avoiding performance degradation (1) By avoiding performance degradation, we mean that R0 does not show a substantial decline on pinhole benchmarks relative to several advanced general-purpose pinhole models. R0 replaces Pi3's point-map head with a ray--radial head, changing the output from three-channel to a four-channel representation. Its weights therefore do not exactly match Pi3 even at initialization, and R0 should not be expected to outperform Pi3 on every pinhole dataset. (2) The original large-scale training experienced several server instabilities and job interruptions. We reran the complete end-to-end training and reevaluated the new checkpoint. The updated results show that R0 reaches state of the art on several pinhole datasets and is not substantially worse than other models on the remaining datasets. The updated pinhole reconstruction, pose-estimation, and depth results are listed below.
(1) Pinhole point-map reconstruction
| Method | 7-Scenes Sparse Acc. ↓ | Comp. ↓ | NC ↑ | 7-Scenes Dense Acc. ↓ | Comp. ↓ | NC ↑ |
|---|---|---|---|---|---|---|
| VGGT | 0.043 | 0.060 | 0.766 | 0.019 | 0.027 | 0.693 |
| Pi3 | 0.045 | 0.071 | 0.747 | 0.013 | 0.016 | 0.694 |
| MapAnything | 0.098 | 0.162 | 0.707 | 0.061 | 0.048 | 0.658 |
| Depth Anything 3 | 0.056 | 0.069 | 0.737 | 0.021 | 0.023 | 0.679 |
| Pi3X | 0.053 | 0.081 | 0.751 | 0.013 | 0.017 | 0.697 |
| VGGT- | 0.040 | 0.052 | 0.746 | 0.029 | 0.030 | 0.673 |
| R0 | 0.044 | 0.071 | 0.731 | 0.015 | 0.012 | 0.676 |
(2) Pinhole camera-pose estimation
| Method | Sintel ATE ↓ | RPE-T ↓ | RPE-R ↓ |
|---|---|---|---|
| Pi3 | 0.074 | 0.039 | 0.279 |
| Pi3X | 0.125 | 0.041 | 0.295 |
| VGGT | 0.172 | 0.062 | 0.471 |
| VGGT- | 0.102 | 0.033 | 0.222 |
| MapAnything | 0.226 | 0.077 | 0.642 |
| Depth Anything 3 | 0.114 | 0.056 | 0.469 |
| R0 | 0.092 | 0.049 | 0.506 |
(3) Pinhole depth estimation
(3a) Monocular depth estimation (per-image scale alignment)
| Method | Structured3D-Pinhole AbsRel ↓ | RMSE ↓ | δ1 ↑ |
|---|---|---|---|
| VGGT | 0.155 | 0.278 | 0.889 |
| Pi3 | 0.480 | 1.017 | 0.895 |
| MapAnything | 0.356 | 0.867 | 0.880 |
| Depth Anything 3 | 0.177 | 0.307 | 0.882 |
| VGGT- | 0.170 | 0.304 | 0.883 |
| Pi3X | 0.237 | 0.505 | 0.901 |
| R0 | 0.128 | 0.281 | 0.912 |
(3b) Video depth estimation (sequence-shared scale and shift)
| Method | Bonn AbsRel ↓ | RMSE ↓ | δ1 ↑ | ScanNet AbsRel ↓ | δ1 ↑ | Sintel AbsRel ↓ | RMSE ↓ | δ1 ↑ |
|---|---|---|---|---|---|---|---|---|
| VGGT | 0.049 | 0.249 | 0.974 | 0.038 | 0.984 | 0.226 | 5.565 | 0.683 |
| Pi3 | 0.044 | 0.221 | 0.975 | 0.026 | 0.990 | 0.210 | 3.978 | 0.728 |
| Pi3X | 0.038 | 0.211 | 0.975 | 0.026 | 0.991 | 0.188 | 4.027 | 0.737 |
| MapAnything | 0.050 | 0.239 | 0.975 | 0.053 | 0.972 | 0.290 | 4.585 | 0.688 |
| Depth Anything 3 | 0.049 | 0.223 | 0.973 | 0.053 | 0.966 | 0.266 | 5.490 | 0.654 |
| VGGT- | 0.115 | 0.349 | 0.932 | 0.036 | 0.980 | 0.356 | 5.740 | 0.625 |
| R0 | 0.041 | 0.206 | 0.976 | 0.025 | 0.992 | 0.217 | 3.704 | 0.685 |
These results cover pinhole reconstruction, pose estimation, monocular depth, and video depth. R0 achieves the best completeness on 7-Scenes Dense, the second-best ATE for Sintel pose estimation, and the best AbsRel and on Structured3D-Pinhole. It also obtains the best RMSE and on Bonn, the best AbsRel and on ScanNet, and the best RMSE for Sintel video depth. Thus, after extending to fisheye and panoramic cameras, R0 remains comparable to or better than advanced pinhole models across multiple pinhole tasks, rather than avoiding degradation on only a single pinhole benchmark. The quantitative results appear in Sections 4.1.1--4.1.3. We also add qualitative visualizations with ERP, pinhole, and fisheye inputs in “Additional reconstruction comparisons” in Appendix A.5, covering different scenes and numbers of input views.
5. Reproducibility
We will release the complete large-scale training code, evaluation code, and model weights to support reproducibility and future research.
Dear Reviewer E2kW,
As the discussion phase is nearing its end, we kindly invite you to review our rebuttal. We have clarified the source of the initial geometry and the cross-view coordinate transformations, added comparisons with camera-specific methods, and revised our discussion of the pinhole-performance trade-off. We would appreciate your feedback and welcome any further questions.
Reply to Reviewer E2kW
W1W2. Additional details of Geometry Attention
(1) Where does the initial geometry used by Geometry Attention come from? R0 uses two coarse-to-fine decoding stages. The first decoding uses only the original attention blocks and predicts initial rays, radial distances, and relative poses through the shared ray--radial and pose heads. The initial 3D points are obtained directly from the rays and radial distances; together, these predictions constitute the coarse geometry required by Geometry Attention. The second decoding reuses the same enhanced image features, introduces Geometry Attention in the later decoder blocks, and passes the result through the same output heads again to obtain the final rays, radial distances, poses, and confidence. We have clarified this consistently in Section 3.3 and Appendix A.1.
Within each view, Geometry Attention uses the initial rays, 3D points, centered radial distances, and valid support in that camera's coordinate frame, allowing image features with similar local geometry to interact. The valid support excludes the exterior of a circular fisheye image, padding, and other locations that do not correspond to physical imaging rays.
(2) How does cross-view interaction handle different camera coordinate frames? For cross-view interaction, the initial 3D points from all views are first transformed into the first-view reference frame using the relative poses predicted by the first decoding. They are then centered and scale-normalized before establishing cross-view geometric correspondences. Rays and radial distances are used primarily for within-view geometric interaction, whereas the transformed 3D points in the shared coordinate frame are used for cross-view interaction. Section 3.3 and Appendix A.1 now explicitly describe the two readings of the output heads and the geometry and coordinate frames used by the two types of Geometry Attention.
W3. Comparison with camera-specific methods We add direct comparisons with CAM3R, Wid3R, and several camera-specific monocular models such as Depth Any Panorama. (1) CAM3R uses a two-view feed-forward model and extends it to multiple views through an unreleased iterative backend. We therefore evaluate its released two-view model for pose estimation and point-map reconstruction on 7-Scenes, FIORD, and OB3D. (2) Wid3R requires the camera type in addition to the images. We use its corresponding built-in camera model for each dataset and compare multi-view reconstruction on OB3D, FIORD, and Deep360. (3) We also compare depth estimation on OB3D and WideDepth with camera-specific monocular models including UniK3D, DAC, DAP, and PanDA. All comparisons use the same test inputs and evaluation protocols.
(1) Two-view relative pose and point-map reconstruction
R0 achieves the best two-view reconstruction results on all three datasets and the best overall relative-pose performance.
(2) Multi-view reconstruction on wide-FoV cameras
We also add comparisons with camera-specific monocular models including UniK3D, DAC, DAP, and PanDA.
(3) Depth estimation comparison with camera-specific monocular models