- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes X-Lens, a feed-forward model for metric depth estimation from a variable number of calibrated fisheye and pinhole views. Each camera is represented by a generic unprojection map that produces unit rays, together with the camera pose and camera-type indicator. The model uses a DINOv2 and DPT backbone, learnable calibration tokens for fisheye adaptation, and a Jacobian Distortion Bias in cross-view attention. The bias is computed from local ray directions and finite-difference unprojection Jacobians on the patch grid. X-Lens predicts a mean-normalized dense depth field, a confidence map, a validity mask, and one global metric scale per sample. A three-stage training scheme first learns from pinhole data, adapts calibration tokens to fisheye inputs, and then jointly trains on heterogeneous data with the Jacobian bias enabled. The paper also introduces OmniScene, a large synthetic six-camera dataset with four fisheye and two pinhole views, and evaluates X-Lens on heterogeneous synthetic data, real fisheye and pinhole benchmarks, and a preliminary robotic downstream task.
Critical strengths
-
The ray-space formulation is well matched to heterogeneous cameras. Unit rays and camera poses preserve native fisheye geometry without panorama conversion.
-
The model separates lens adaptation from cross-view interaction. View-local calibration tokens adapt fisheye features, while the Jacobian bias acts in global attention.
-
The factored output suits a compact metric model. Normalized depth and one metric-scale scalar reduce the output burden, while confidence-weighted pooling limits the influence of unreliable boundary regions.
Critical weaknesses
-
The central heterogeneous-camera accuracy claim lacks quantitative real-world validation. Joint fisheye–pinhole benchmark results use synthetic OmniScene, and real-world depth benchmarks use homogeneous inputs. A five-camera robot demonstration shows deployment feasibility, but does not report ground-truth depth accuracy for mixed-camera scenes under real appearance, synchronization, calibration, and sensor noise.
-
The JDB bias does not explicitly model cross-view ray correspondence. Its descriptor uses local rays and Jacobians, while camera origins enter through the broader ray map; it has no explicit epipolar or ray-intersection relation. Its behavior with large baselines, limited overlap, or repeated ray directions is therefore unclear.
-
The method requires known calibration and a usable unprojection map at inference. It does not estimate intrinsics, extrinsics, distortion, or the unprojection function online, so its performance depends on the accuracy of these inputs.
-
The effect of heterogeneous view reliability on global scale is not fully characterized. Depth is normalized jointly and one scalar is predicted per sample, but the paper does not report per-view scale consistency or the effect of noisy, invalid, or weakly overlapping views on this estimate.
-
The comparison omits cited panorama-based methods, including Depth Anything in 360 (Jiang et al., 2025) and Depth Any Panoramas (Lin et al., 2026), making the advantage over panorama-based processing harder to assess.
Questions for authors
-
Because confidence-weighted spatial features are pooled into one global scale, how does the model handle conflicting scale evidence across views with little overlap or many invalid regions?
-
How does JDB identify corresponding patches with large baselines or repeated ray directions, and is camera-origin information in the general ray map sufficient without an explicit epipolar or ray-intersection constraint?
4 comments
Author Response [2/2]
W3: Dependence on known calibration
Thank you for the comment. We also acknowledge this limitation. X-Lens targets pre-calibrated rigs and does not estimate intrinsics, extrinsics, or distortion online. We kept calibration as an input because predicting it jointly hurt depth accuracy in our compact model. Appendix E.2 (Table 14) adds camera tokens with an intrinsics/extrinsics head and a ray-map DPT head to the Stage-1 model, under the same data, schedule, and parameter budget:
| Variant (six-view OmniOcc, after Stage 1) | Scale AbsRel ↓ | AbsRel ↓ | RMSE ↓ | ↑ | ↑ |
|---|---|---|---|---|---|
| With camera head + ray-map head | 0.1816 | 0.0702 | 0.2861 | 0.8961 | 0.4910 |
| Depth, confidence, and scale heads only (ours) | 0.0692 | 0.0636 | 0.2499 | 0.9481 | 0.5442 |
With the auxiliary camera prediction, all depth metrics degrade, and Scale AbsRel rises from 0.0692 to 0.1816. Appendix D.4 further evaluates robustness to perturbations of the fisheye intrinsics. The conclusion of the revised paper now points to this limitation, which Appendix G discusses together with online calibration refinement as future work.
(Revised in the paper: Lines 498–502 in Section 7 and Lines 1511–1514 in Appendix E.2. Table 14 is unchanged.)
W4 & Q1: Conflicting scale evidence and per-view reliability
X-Lens predicts one scale factor shared by all views in a sample, which avoids inconsistencies between independently estimated per-view scales. Validity masks exclude sky and other invalid regions from supervision, and confidence-weighted pooling reduces the contribution of unreliable regions to the global scale.
These mechanisms downweight unreliable evidence, but they cannot create cross-view evidence that is not there. When views overlap very little or contain large invalid regions, scale accuracy can drop. The KITTI360 experiment above illustrates this. We acknowledge that a systematic analysis of per-view scale consistency and of the sensitivity of the shared scale to noisy, invalid, or weakly overlapping views is missing, and Appendix G now lists it as a limitation.
(Revised in the paper: Lines 1618–1620 in Appendix G and Lines 1875–1881 in Appendix H.3, with Table 19.)
W5: Panorama-based methods
Thank you for this suggestion. Depth Anything in 360 and Depth Any Panoramas estimate depth from panoramic images, whereas X-Lens jointly processes separate native fisheye and pinhole views from a calibrated rig. Our claim concerns joint depth estimation from these heterogeneous inputs, not improved depth estimation on panoramas. A direct comparison would require stitching the original views into a panorama first, so it would evaluate the combined stitching and depth-estimation pipeline.
As a related data point, we evaluated X-Lens zero-shot on 2,014 native Matterport3D-360 panoramas:
| Method | Params | Scale AbsRel ↓ | AbsRel ↓ | RMSE ↓ | ↑ | ↑ |
|---|---|---|---|---|---|---|
| DAC | 0.06B | 0.1581 | 0.2065 | 1.9571 | 0.6475 | 0.1701 |
| UniDAC | 0.36B | 0.1663 | 0.1592 | 1.8933 | 0.6851 | 0.1789 |
| X-Lens | 0.04B | 0.2030 | 0.3755 | 2.4781 | 0.5512 | 0.1407 |
X-Lens, which never observes equirectangular inputs during training, trails DAC and UniDAC here. X-Lens should therefore not be read as a panoramic depth estimator. These results are in Appendix H.4 (Table 20, Figure 16) of the revised PDF. We have added this distinction to the Limitations (Appendix G).
(Revised in the paper: Lines 1611–1615 in Appendix G and Lines 1883–1938 in Appendix H.4, with new Table 20 and Figure 16.)
Thank you very much for the detailed response and additional analyses. After careful consideration, I will keep my original score.
Thank you for taking the time and effort to review our paper. We greatly appreciate your constructive feedback, which has helped us improve our work.
Author Response [1/2]
We sincerely thank the reviewer for the careful and insightful review. We are glad the reviewer found the ray-space formulation well matched to heterogeneous cameras, the separation of lens adaptation from cross-view interaction sound, and the factored output suitable for a compact metric model. Below, we first give a brief overview of our responses, followed by detailed clarifications.
W1: Quantitative real-world heterogeneous validation
We acknowledge this weakness, which is discussed in Appendix G. Our five-camera robot rig has no LiDAR, so quantitative evaluation on it was unfortunately not possible. We instead evaluate joint inference with two fisheye views and one pinhole view on real-world KITTI360 sequence 0, with LiDAR ground truth:
X-Lens outperforms DAC and approaches UniDAC with 89% fewer parameters, but does not surpass it. The fisheye and pinhole cameras of KITTI360 barely overlap, so this setting tests the regime where joint inference has the least shared evidence, rather than its full benefit. These results are in Appendix H.3 (Table 19), with qualitative results in Figure 15 of the revised PDF. A heterogeneous real-world benchmark with substantial overlap and dense ground truth remains missing, and Appendix G states this explicitly.
(Revised in the paper: Lines 420–422 in Section 5, Lines 1603–1606 in Appendix G, and Lines 1858–1881 in Appendix H.3, with new Table 19 and Figure 15.)
W2 & Q2: JDB with large baselines, limited overlap, or repeated ray directions
The JDB descriptor combines four terms: angular similarity, relative local projection scale, ray-direction differences, and Jacobian correlations that capture local orientation and anisotropic distortion. An MLP maps these terms to a bias that is added to the query–key attention scores. JDB applies no hard angular matching threshold. Corresponding patches under large baselines can therefore still attend to each other even when their ray directions differ substantially, because the final attention depends on both visual features and the geometric bias.
When ray directions repeat across cameras, the normalized camera origins in the general ray map provide additional information to the queries and keys. Together with appearance, this helps distinguish patches observed along the same direction from different positions. Origin and direction define each ray in a common normalized frame, but providing this information does not enforce epipolar consistency or ray intersection; the model learns to use these cues during joint training. JDB is therefore a soft prior. We agree that ambiguities can remain when overlap is limited or appearance is repetitive, and the KITTI360 results in W1 show this degradation.
Section 3.2 now describes JDB as a soft geometric prior on cross-view attention and points to Appendix G. Appendix G states that JDB imposes no epipolar or ray-intersection constraint, so cross-view matching may remain ambiguous under limited overlap, large baselines, or repetitive appearance.
(Revised in the paper: Lines 304–306 in Section 3.2 and Lines 1616–1618 in Appendix G.)