OmniRay: Monocular 3D Reconstruction with Composite Ray Fields

Monocular models recover 3D geometry from a single image in one pass, but most assume a pinhole camera and produce distorted geometry on the wide-angle imagery produced by fisheye and panoramic lenses. Camera-aware models re- move this assumption by predicting a per-pixel ray field. However, they represent different camera geometries using a single fixed basis to capture both the overall camera projection and its finer variations. We argue that camera geometry has a hierarchy of complexity and that the ray representation should follow it. We present OmniRay, a feed-forward model whose ray module is composed of three components: a monotone radial base that captures radially symmetric structure, a spherical-harmonic residual for smooth deviations from it, and a free-form resid- ual for remaining local variation. These components are composed as successive rotations on the sphere, allowing the representation to accommodate diverse cam- era geometries. Given a single image, OmniRay predicts a ray direction and an up- to-scale radial distance for every pixel, from which a 3D pointcloud is recovered without needing camera information at inference time. Extensive experiments across different camera configurations demonstrate that OmniRay establishes a new state-of-the-art in camera-agnostic monocular 3D reconstruction.