acceptodds
Under review as a conference paper at ICLR 2027

OmniRay: Monocular 3D Reconstruction with Composite Ray Fields

Abstract

Monocular models recover 3D geometry from a single image in one pass, but most assume a pinhole camera and produce distorted geometry on the wide-angle imagery produced by fisheye and panoramic lenses. Camera-aware models re- move this assumption by predicting a per-pixel ray field. However, they represent different camera geometries using a single fixed basis to capture both the overall camera projection and its finer variations. We argue that camera geometry has a hierarchy of complexity and that the ray representation should follow it. We present OmniRay, a feed-forward model whose ray module is composed of three components: a monotone radial base that captures radially symmetric structure, a spherical-harmonic residual for smooth deviations from it, and a free-form resid- ual for remaining local variation. These components are composed as successive rotations on the sphere, allowing the representation to accommodate diverse cam- era geometries. Given a single image, OmniRay predicts a ray direction and an up- to-scale radial distance for every pixel, from which a 3D pointcloud is recovered without needing camera information at inference time. Extensive experiments across different camera configurations demonstrate that OmniRay establishes a new state-of-the-art in camera-agnostic monocular 3D reconstruction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Monocular 3D estimation, Camera-Agnostic, Ray Representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XCv8OpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses monocular 3D reconstruction for central cameras, aiming to handle pinhole, fisheye, catadioptric and panoramic images without intrinsics or a camera-type label. The authors argue that a single fixed spherical-harmonic basis, as used in UniK3D, is a poor fit for all cameras, and split the ray field into three layers. A 16-knot monotone radial profile maps image radius to incidence angle, a degree-7 spherical-harmonic residual is added on top, and a convolutional decoder supplies a per-pixel free-form residual. Both residuals are applied as rotations on the sphere. A DINOv2 and DPT distance branch predicts per-pixel radial distance, which is multiplied with the ray to obtain the point cloud. The experiments start with an oracle study that fits each representation directly to ground-truth rays. The model is then trained on 2D3DS and 360Loc and compared with UniK3D, DA3, GeoCalib and AnyCalib on seen cameras, on held-out catadioptric and GoPro cameras, and on the PanoCity and CO3Dv2 datasets, followed by an ablation of the ray representation.

Critical strengths

The layered ray representation is a reasonable idea. The oracle study in Table 1 shows that a radial model cannot express equirectangular projection, that spherical harmonics lack precision on fisheye, and that the composition fits all three geometries to within 0.25 degrees. This analysis is clear and is the most valuable part of the paper. On strongly distorted catadioptric and panoramic images, the numerical advantage over existing baselines is also large.

Critical weaknesses

My main concern is that the comparisons are not fair enough to support the central claims. OmniRay is trained on synthetic lenses rendered from panoramas, and the random-lens augmentation already covers annular image domains. The held-out cameras are kept unseen only by a rejection rule that is not described in the main text. Among the baselines, only UniK3D is fine-tuned on the same data; DA3 is not, and GeoCalib and AnyCalib only perform calibration. The gap on seen cameras, 0.14 degrees against 6.34 degrees, is essentially an in-distribution versus out-of-distribution comparison. All baselines exceed 75 degrees of error on the catadioptric camera, which tells me they have simply never seen such inputs, so this result says little about the representation itself. Recent camera-aware methods such as DAC, UniDAC, MoGe-2, Wid3R and CAM3R are also missing. The claim of a new state of the art by a large margin is therefore not yet supported.

Second, the method is not better on ordinary images, and the ablations are thin. On CO3Dv2 pinhole images, OmniRay obtains 2.97 degrees ray error and 0.038 Chamfer distance, clearly worse than DA3 at 2.04 degrees and 0.028. This contradicts the statement that flexibility for wide-angle cameras does not come at the expense of pinhole imagery. Training uses only two panorama datasets, so scene coverage is limited. The ablation in Table 3 is run on a small held-out subset with coarsely rounded numbers and no variance, and adding the free-form residual makes GoPro worse, from 5.00 to 5.50 degrees. In Table 1 the composite is also worse than the radial base alone on fisheye and catadioptric cameras, 0.075 against 0.002 and 0.174 against 0.023. The paper does not explain when the composition helps and when it hurts.

Questions for authors
  1. What exactly is the rejection rule that keeps the catadioptric and GoPro cameras out of the random-lens augmentation, and how close are the nearest accepted training lenses to these test cameras?
  2. What happens if UniK3D is trained with the same random-lens augmentation? This experiment is essential to tell whether the gain comes from the representation or from the training data.
  3. Why is the composite worse than the radial base on fisheye and catadioptric in Table 1, and why does the free-form residual hurt GoPro in Table 3? Does the spherical-harmonic residual interfere with a radial base that should already suffice?
  4. Could you add depth metrics such as AbsRel and δ1 to the main table, and compare with DAC, UniDAC and MoGe-2 on public benchmarks such as ScanNet++ fisheye and KITTI-360?
1 comment

Thank you for the additional experiments and explanations. The comparisons using the same augmentation and the ablations on the full test set have eased my concerns about fairness and whether the individual components help. However, after reading the revised results, I still have reservations about the significance of the contribution and lean toward rejection.The paper combines a radial model, spherical harmonics, and a per-pixel residual into a layered ray prediction module. This design has some novelty, and the additional experiments support its effectiveness. My concern is not that using existing components makes a method unoriginal. Rather, I am not yet convinced that the improved representation brings a sufficiently meaningful gain in 3D reconstruction. The ablations show that the components complement each other, but they do not fully establish how much this matters for practical reconstruction.

The largest gains still come from synthetic-camera tests, particularly the catadioptric camera. The new real-data experiments provide some support, but the reconstruction evidence remains limited. ScanNet++ evaluates ray error only. After both methods are fine-tuned on ADT, CD decreases from UniK3D’s 0.0569 to 0.0552, a relatively small improvement. The CO3Dv2 pinhole results also show no clear reconstruction advantage. I acknowledge that PanoCity provides an outdoor street-scene evaluation. Still, the evidence for better reconstruction under real imaging conditions is less convincing than the evidence for better ray estimation on synthetic cameras. My remaining concern is therefore about the scope and significance of the demonstrated gains, rather than the absence of a particular dataset or baseline.

I also have one question about the rejection rule. For GoPro, the rule ensures that a training lens differs from the test camera by more than 15° at at least one radius. It does not imply that the mean ray-angle difference exceeds 15°. The catadioptric rule instead constrains the inner and rim angles. Please clarify what distance measure “at least 15° away” refers to, and whether you measured the nearest distance between the actual training samples and each test configuration. I agree with calling these tests “held-out camera configurations,” but the description of their geometric separation should be precise.

Reviewer_q6UDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies recovering up-to-scale 3D geometry from a single image without camera intrinsics or a camera-type label. The authors propose a composite ray field representation: a monotone radial base captures the radially symmetric structure, a spherical-harmonic residual expresses smooth deviations from it, and a free-form residual accounts for the remaining local variation. The three components are composed as successive rotations on the sphere S² (applied through the exponential map). A Distance Module predicts a radial distance ρ for every pixel, and the camera-frame pointcloud is obtained by scaling each ray by its distance, P = ρd.

The authors train on two panoramic datasets, 2D3DS and 360Loc, with Random-lens Augmentation, and evaluate on held-out scenes, held-out cameras (a GoPro fisheye and a Nayar catadioptric camera), and two datasets unseen in training (held-out domains: PanoCity and CO3Dv2). Validation includes an oracle analysis that fits each representation directly to ground-truth rays (oracle test: representation ceilings), benchmark comparisons against existing methods, and ablations of the ray representation and the data augmentation. The reported results show clear improvements in the wide-angle and panoramic settings tested.

Critical strengths
  1. The hierarchical ray representation has a sound geometric inductive bias. The monotone radial base suits the dominant projection structure of conventional central cameras, while the global spherical-harmonic residual and the local free-form residual provide flexibility for departures from that structure. Compared with using a single fixed basis to capture both the overall camera projection and its finer variations, as in UniK3D, this design offers a worthwhile structural choice for camera-agnostic monocular reconstruction. The specific contribution lies in the composition of the three representation layers, not in the decomposition into a ray direction and a radial distance itself.

  2. The reported gains in wide-angle settings are substantial and supported by multiple metrics and qualitative results. In Table 2, on seen cameras the mean angular error (Ang.) drops from UniK3D's 6.336° to 0.140°, and the relative Chamfer distance (CD) from 0.0760 to 0.0179; on the held-out catadioptric camera the error drops from 75.702° to 1.120°; PanoCity also improves markedly (21.220° → 1.350°). The panoramic and catadioptric examples in Figure 4 show more complete geometry. These results indicate real practical potential.

Critical weaknesses
  1. The independent contribution of the core representation design is not yet adequately validated. Section 4.3 states that UniK3D is fine-tuned on the 2D3DS and 360Loc panoramas "so that it sees the same source imagery we train on," but it does not say whether UniK3D uses exactly the same Random-lens Augmentation, sampling distribution, and training budget as OmniRay. The same source imagery does not imply the same training input distribution. Meanwhile, in Table 2, adding the random-lens augmentation alone (w/o AUG → full model) reduces the catadioptric angular error from 30.867° to 1.120° and CD from 0.3440 to 0.0131, showing that the augmentation is an extremely influential factor. Table 3 (ray representation ablation) evaluates rays only on a small held-out subset of 2D3DS, and it additionally mixes joint and staged training regimes; key controls under identical training conditions — such as radial+free-form and SH+free-form — are missing. It is therefore difficult to judge whether the large gains come mainly from the three-layer composite representation, the Random-lens Augmentation, or the training strategy. This directly affects the assessment of the paper's core technical contribution.
Questions for authors
  1. Fair-comparison controls for validating the core contribution: Do all baselines and ablations share OmniRay's training conditions? Specifically, does the fine-tuned UniK3D use the same Random-lens Augmentation sampling distribution, the same rejection criteria near held-out geometries, the same input resolution, and the same training budget? Please specify its checkpoint and fine-tuning setup, as well as which weights and which camera fitting model AnyCalib uses. Section 4.3 says the estimated cameras of GeoCalib/AnyCalib are converted to dense ray fields for evaluation ("convert their estimated cameras to dense ray fields"); please clarify whether the evaluation uses their direct predictions or rays regenerated after fitting a parametric camera model. More importantly, are there control results under identical Random-lens Augmentation and identical training strategy (consistent joint/staged) for SH-only, radial+free-form, and the full three-layer model (radial+SH+free-form)? If available, please provide both ray (Ang.) and pointcloud (CD) metrics. These controls directly affect the interpretation of the main performance gaps and the judgment of whether the monotone radial base and the intermediate spherical-harmonic residual layer are necessary; they are evidence needed to validate the current contribution, not suggestions for future work.

  2. Clarification of the oracle analysis: Do the fits in Table 1 have any guarantee of global optimality or sufficient convergence? How do you explain that the composite model containing the radial base (Radial16+SH7) obtains a larger fitting error on the fisheye (0.075° vs 0.002°)? If the composite model is initialized from the already-fitted radial solution with the residuals zero-initialized (as done in training), how do the results change? Please clarify accordingly which conclusions are empirical fitting results and which carry guarantees at the level of representation capacity.

  3. Clarification of the claim scope: Can you provide results on the perspective (pinhole) subset of CO3Dv2 to support the statement that pinhole performance is not sacrificed? Meanwhile, the sampling family of the Random-lens Augmentation already covers filled and annular domains and monotone radial profiles, and only samples close to the held-out geometries are rejected by the rejection criteria; should the catadioptric test therefore be described more accurately as generalization to a held-out camera configuration, rather than an entirely unseen lens family? These clarifications will affect the assessment of the applicable scope and the generalization contribution.

Reviewer_jae7OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method for joint estimation of camera intrinsics (under unknown camera assumption) and monocular depth estimation. The method is based on a hierarchical modeling of the camera ray field, based on a base radial model, a spherical harmonics residual model, and a free-form residual correction. The other part of the model then predicts distance along the predicted ray direction to provide depth. The method shows promising results in generalizing to multiple different camera models, and even unseen camera models, achieveing better average ray direction error, and also improving on depth estimation.

Critical strengths

The paper proposes an explicit modeling of camera intrinsics in the monocular depth estimation pipeline, that is shown to: 1- generalize to different camera models without known camera 2- improve the depth estimation error this is very useful for scaling depth estimation to general inputs from different camera models. The training recipe is also interesting as this can take advantage of arbitrary augmentations on the ray field.

Critical weaknesses

My main critique is with the ablation table 3, which supports one of the main contributions of the paper i.e. the need to model intrinsics in a hierarchical fashion. This table shows that by adding each part of the hierarchy in the intrinsic modeling you gain added performance. However, I'm more interested in knowing how much the first two levels of hierarchy (the inductive bias) are needed and why would a free-from residual layer alone not be able to model the rayfield completely. Therefore my main request is to have an ablation table that at each time it drops 1 component from your pipeline and keeps the rest in tact so I can access how much each level of the hierarchy, alone, was contributing to the performance. Additionally, the results provided in this table are not reporting Chamfer Distance, could you also please add that?

Other minor concerns:

  • What is LregionL_{region} in equation 7?
  • I personally associate monocular reconstruction more with methods like Zero-1-to-3 [1], which aim to reconstruct a complete scene or object from a monocular image or video. I understand why the term reconstruction is being used here, especially given the recent popularity of pointmap prediction methods. However, since this work operates on a single image rather than a video, I would argue that it is more closely aligned with monocular depth estimation and should be named accordingly.
  • Figure 5 (b) is hard to interprete, can you please remove some of the points in the front that obstruct the view so its easy to judge the quality of the produced geometry?

[1] Zero-1-to-3: Zero-shot One Image to 3D Object, Liu et al, CVPR 2023

Questions for authors

No more questions except the above ask for the completed ablation table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.