acceptodds
Under review as a conference paper at ICLR 2027

X-Lens: Real-Time Metric Depth Estimation with Heterogeneous Cameras

Abstract

We present X-Lens, a compact feed-forward model for metric depth estimation from a variable number of distorted and pinhole views. To support real-time downstream perception, X-Lens is built around a geometry-aware heterogeneous camera formulation with two key components. Learnable calibration tokens provide a coarse alignment between fisheye and pinhole projective spaces, while a Jacobian-parameterized distortion bias injected into cross-attention models local projection changes and promotes cross-camera consistency, enabling robust generalization with only 0.04B parameters and up to 41 FPS. The model predicts dense depth together with a global metric scale, avoiding auxiliary reconstruction targets that increase computation and optimization complexity. To facilitate cross-camera generalization at scale and depth, X-Lens is trained on multiple public datasets and OmniScene, our newly curated large-scale synthetic dataset containing 266K synchronized six-view frames, around 1.6M individual images, and 126 indoor and outdoor scenes. Extensive experiments demonstrate superior heterogeneous-camera metric depth accuracy on our OmniScene dataset, alongside optimal or competitive performance on real-world fisheye and pinhole-only benchmarks. On OmniScene-Full, X-Lens reduces AbsRel by 25.4% over the strongest baseline while using 88.9% fewer parameters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Depth Estimation, Feed Forward

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dqa9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes X-Lens, a feed-forward model for metric depth estimation from a variable number of calibrated fisheye and pinhole views. Each camera is represented by a generic unprojection map that produces unit rays, together with the camera pose and camera-type indicator. The model uses a DINOv2 and DPT backbone, learnable calibration tokens for fisheye adaptation, and a Jacobian Distortion Bias in cross-view attention. The bias is computed from local ray directions and finite-difference unprojection Jacobians on the patch grid. X-Lens predicts a mean-normalized dense depth field, a confidence map, a validity mask, and one global metric scale per sample. A three-stage training scheme first learns from pinhole data, adapts calibration tokens to fisheye inputs, and then jointly trains on heterogeneous data with the Jacobian bias enabled. The paper also introduces OmniScene, a large synthetic six-camera dataset with four fisheye and two pinhole views, and evaluates X-Lens on heterogeneous synthetic data, real fisheye and pinhole benchmarks, and a preliminary robotic downstream task.

Critical strengths
  1. The ray-space formulation is well matched to heterogeneous cameras. Unit rays and camera poses preserve native fisheye geometry without panorama conversion.

  2. The model separates lens adaptation from cross-view interaction. View-local calibration tokens adapt fisheye features, while the Jacobian bias acts in global attention.

  3. The factored output suits a compact metric model. Normalized depth and one metric-scale scalar reduce the output burden, while confidence-weighted pooling limits the influence of unreliable boundary regions.

Critical weaknesses
  1. The central heterogeneous-camera accuracy claim lacks quantitative real-world validation. Joint fisheye–pinhole benchmark results use synthetic OmniScene, and real-world depth benchmarks use homogeneous inputs. A five-camera robot demonstration shows deployment feasibility, but does not report ground-truth depth accuracy for mixed-camera scenes under real appearance, synchronization, calibration, and sensor noise.

  2. The JDB bias does not explicitly model cross-view ray correspondence. Its descriptor uses local rays and Jacobians, while camera origins enter through the broader ray map; it has no explicit epipolar or ray-intersection relation. Its behavior with large baselines, limited overlap, or repeated ray directions is therefore unclear.

  3. The method requires known calibration and a usable unprojection map at inference. It does not estimate intrinsics, extrinsics, distortion, or the unprojection function online, so its performance depends on the accuracy of these inputs.

  4. The effect of heterogeneous view reliability on global scale is not fully characterized. Depth is normalized jointly and one scalar is predicted per sample, but the paper does not report per-view scale consistency or the effect of noisy, invalid, or weakly overlapping views on this estimate.

  5. The comparison omits cited panorama-based methods, including Depth Anything in 360 (Jiang et al., 2025) and Depth Any Panoramas (Lin et al., 2026), making the advantage over panorama-based processing harder to assess.

Questions for authors
  1. Because confidence-weighted spatial features are pooled into one global scale, how does the model handle conflicting scale evidence across views with little overlap or many invalid regions?

  2. How does JDB identify corresponding patches with large baselines or repeated ray directions, and is camera-origin information in the general ray map sufficient without an explicit epipolar or ray-intersection constraint?

4 comments
AuthorsOpenReview ↗

Author Response [1/2]

We sincerely thank the reviewer for the careful and insightful review. We are glad the reviewer found the ray-space formulation well matched to heterogeneous cameras, the separation of lens adaptation from cross-view interaction sound, and the factored output suitable for a compact metric model. Below, we first give a brief overview of our responses, followed by detailed clarifications.

  • Real-world heterogeneous validation. We add a quantitative joint fisheye–pinhole evaluation on real-world KITTI360 with LiDAR ground truth. This setting also shows the limited-overlap regime the reviewer asks about.
  • JDB and global scale. JDB is a soft geometric prior, not a correspondence constraint. Confidence weighting cannot compensate for missing cross-view evidence. We acknowledge both limitations explicitly.
  • Calibration and panorama baselines. We clarify the reliance on known calibration and the relation to panorama-based methods, supported by a new preliminary panoramic evaluation.

W1: Quantitative real-world heterogeneous validation

We acknowledge this weakness, which is discussed in Appendix G. Our five-camera robot rig has no LiDAR, so quantitative evaluation on it was unfortunately not possible. We instead evaluate joint inference with two fisheye views and one pinhole view on real-world KITTI360 sequence 0, with LiDAR ground truth:

MethodParamsScale AbsRel ↓AbsRel ↓RMSE ↓δ1\delta_1 ↑τ1.03\tau_{1.03} ↑
DAC0.06B0.20250.27784.51240.56410.1137
UniDAC0.36B0.16720.22943.50770.73320.1588
X-Lens0.04B0.17310.23313.92520.69180.1401

X-Lens outperforms DAC and approaches UniDAC with 89% fewer parameters, but does not surpass it. The fisheye and pinhole cameras of KITTI360 barely overlap, so this setting tests the regime where joint inference has the least shared evidence, rather than its full benefit. These results are in Appendix H.3 (Table 19), with qualitative results in Figure 15 of the revised PDF. A heterogeneous real-world benchmark with substantial overlap and dense ground truth remains missing, and Appendix G states this explicitly.

(Revised in the paper: Lines 420–422 in Section 5, Lines 1603–1606 in Appendix G, and Lines 1858–1881 in Appendix H.3, with new Table 19 and Figure 15.)


W2 & Q2: JDB with large baselines, limited overlap, or repeated ray directions

The JDB descriptor combines four terms: angular similarity, relative local projection scale, ray-direction differences, and Jacobian correlations that capture local orientation and anisotropic distortion. An MLP maps these terms to a bias that is added to the query–key attention scores. JDB applies no hard angular matching threshold. Corresponding patches under large baselines can therefore still attend to each other even when their ray directions differ substantially, because the final attention depends on both visual features and the geometric bias.

When ray directions repeat across cameras, the normalized camera origins in the general ray map provide additional information to the queries and keys. Together with appearance, this helps distinguish patches observed along the same direction from different positions. Origin and direction define each ray in a common normalized frame, but providing this information does not enforce epipolar consistency or ray intersection; the model learns to use these cues during joint training. JDB is therefore a soft prior. We agree that ambiguities can remain when overlap is limited or appearance is repetitive, and the KITTI360 results in W1 show this degradation.

Section 3.2 now describes JDB as a soft geometric prior on cross-view attention and points to Appendix G. Appendix G states that JDB imposes no epipolar or ray-intersection constraint, so cross-view matching may remain ambiguous under limited overlap, large baselines, or repetitive appearance.

(Revised in the paper: Lines 304–306 in Section 3.2 and Lines 1616–1618 in Appendix G.)

AuthorsOpenReview ↗

Author Response [2/2]

W3: Dependence on known calibration

Thank you for the comment. We also acknowledge this limitation. X-Lens targets pre-calibrated rigs and does not estimate intrinsics, extrinsics, or distortion online. We kept calibration as an input because predicting it jointly hurt depth accuracy in our compact model. Appendix E.2 (Table 14) adds camera tokens with an intrinsics/extrinsics head and a ray-map DPT head to the Stage-1 model, under the same data, schedule, and parameter budget:

Variant (six-view OmniOcc, after Stage 1)Scale AbsRel ↓AbsRel ↓RMSE ↓δ1\delta_1 ↑τ1.03\tau_{1.03} ↑
With camera head + ray-map head0.18160.07020.28610.89610.4910
Depth, confidence, and scale heads only (ours)0.06920.06360.24990.94810.5442

With the auxiliary camera prediction, all depth metrics degrade, and Scale AbsRel rises from 0.0692 to 0.1816. Appendix D.4 further evaluates robustness to perturbations of the fisheye intrinsics. The conclusion of the revised paper now points to this limitation, which Appendix G discusses together with online calibration refinement as future work.

(Revised in the paper: Lines 498–502 in Section 7 and Lines 1511–1514 in Appendix E.2. Table 14 is unchanged.)


W4 & Q1: Conflicting scale evidence and per-view reliability

X-Lens predicts one scale factor shared by all views in a sample, which avoids inconsistencies between independently estimated per-view scales. Validity masks exclude sky and other invalid regions from supervision, and confidence-weighted pooling reduces the contribution of unreliable regions to the global scale.

These mechanisms downweight unreliable evidence, but they cannot create cross-view evidence that is not there. When views overlap very little or contain large invalid regions, scale accuracy can drop. The KITTI360 experiment above illustrates this. We acknowledge that a systematic analysis of per-view scale consistency and of the sensitivity of the shared scale to noisy, invalid, or weakly overlapping views is missing, and Appendix G now lists it as a limitation.

(Revised in the paper: Lines 1618–1620 in Appendix G and Lines 1875–1881 in Appendix H.3, with Table 19.)


W5: Panorama-based methods

Thank you for this suggestion. Depth Anything in 360 and Depth Any Panoramas estimate depth from panoramic images, whereas X-Lens jointly processes separate native fisheye and pinhole views from a calibrated rig. Our claim concerns joint depth estimation from these heterogeneous inputs, not improved depth estimation on panoramas. A direct comparison would require stitching the original views into a panorama first, so it would evaluate the combined stitching and depth-estimation pipeline.

As a related data point, we evaluated X-Lens zero-shot on 2,014 native Matterport3D-360 panoramas:

MethodParamsScale AbsRel ↓AbsRel ↓RMSE ↓δ1\delta_1 ↑τ1.03\tau_{1.03} ↑
DAC0.06B0.15810.20651.95710.64750.1701
UniDAC0.36B0.16630.15921.89330.68510.1789
X-Lens0.04B0.20300.37552.47810.55120.1407

X-Lens, which never observes equirectangular inputs during training, trails DAC and UniDAC here. X-Lens should therefore not be read as a panoramic depth estimator. These results are in Appendix H.4 (Table 20, Figure 16) of the revised PDF. We have added this distinction to the Limitations (Appendix G).

(Revised in the paper: Lines 1611–1615 in Appendix G and Lines 1883–1938 in Appendix H.4, with new Table 20 and Figure 16.)

Thank you very much for the detailed response and additional analyses. After careful consideration, I will keep my original score.

Thank you for taking the time and effort to review our paper. We greatly appreciate your constructive feedback, which has helped us improve our work.

Reviewer_CNtAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified monocular depth framework for heterogeneous cameras, with a general ray representation, calibration tokens, Jacobian Distortion Bias, three-stage training, and factored depth/scale output. The engineering solution is fairly complete, notable for its 0.04B parameters and edge-deployment results, and it introduces the OmniScene dataset with multi-dataset and downstream validation. However, the weaknesses are material. "Heterogeneous cameras" in practice covers mainly pinhole and fisheye despite the broader framing. Generalization is claimed but under-verified: models are trained and mainly tested on OmniScene and KITTI360, with no real-world heterogeneous fisheye–pinhole joint evaluation and no domain-gap or sim-to-real failure analysis. The multitasking architecture is also not systematically validated—no task-head removal ablations or analysis of effects on intermediate features. The authors' questions (one-stage vs. three-stage necessity, the scale head design, the UnidepthV2 accuracy drop in Table 1, fine-tuning the zero-shot DepthAnyCamera, and "scale AbsRel" evaluation) are well-posed and could materially affect the assessment. Overall, this is a solid engineering contribution with a useful dataset, but its generalization claims are under-verified and architectural choices lack systematic justification. I lean toward weak rejection.

Critical strengths
  1. The general ray representation, calibration tokens, Jacobian Distortion Bias, three-stage training, and factored depth/scale output form a relatively complete engineering solution. The 0.04B parameters and edge deployment results are notable highlights.
  2. Including validation results from multiple datasets targeting the research question, as well as downstream validation experiments.
  3. Propose a high-quality dataset OmniScene for training and evaluation.
Critical weaknesses
  1. “Heterogeneous cameras” contains telephoto, RGB-D, pinhole, fisheye cameras, and so on. The paper mainly investigates the combination of pinhole and fisheye.
  2. Generalization is mentioned in introduction but insufficiently verified. Authors train model on OmniScene and KITTI360 datasets and mainly test on them.
  3. The interaction between tasks in multitasking architecture has not been systematically validated. For example, gradual removal of various task heads, as well as the optimization effect of multitasking on intermediate features.
Questions for authors
  1. Can authors supplement the real-world heterogeneous fisheye pinhole joint quantitative evaluation?
  2. What are the specific indicators that reflect the domain gap between OmniScene and real data? Are there any failure cases and sim to real analysis?
  3. Can the whole framework be trained in only 1 stage? What is the necessity of designing a three-stage training?
  4. Why do authors adopt an additional scale head instead of setting specific constraints on the approximately mean value of depth output?
  5. What is the reason for the decrease in accuracy of Unidepthv2 using a larger model in Table 1?
  6. DepthAnyCamera seems a zero-shot method dealing with different camera configurations. Why do authors fine-tune this model on OmniScene? Can multi-view methods like VGGT be fine-tuned on OmniScene?
  7. Are there more researches using “scale AbsRel” for depth evaluation?
7 comments
AuthorsOpenReview ↗

Author Response [1/4]

We sincerely thank the reviewer for the detailed and constructive comments. We are glad the reviewer found the overall solution complete, the 0.04B model and edge deployment notable, and OmniScene a high-quality dataset. Below, we first give a brief overview of our responses, followed by detailed clarifications.

  • Real-world heterogeneous evaluation. We add a quantitative and qualitative joint fisheye–pinhole evaluation on real-world KITTI360 (two fisheye + one pinhole). We also explain why real heterogeneous benchmarks with sufficient view overlap remain scarce.
  • Generalization and sim-to-real. X-Lens is evaluated zero-shot on real-world ETH3D and OmniOcc, which are not used for training. We identify limited cross-view overlap as the main failure case and illustrate it on KITTI360.
  • Scope. We narrow "heterogeneous cameras" to fisheye–pinhole rigs in the revision and add a preliminary panoramic study.
  • Design choices. We justify the three-stage training and the factored scale head with existing ablations (Tables 3 and 12). We also clarify the UniDepthV2 trend, the DAC fine-tuning, and the Scale AbsRel metric.

W1: Scope of "heterogeneous cameras"

Thank you for pointing this out. We agree that our study covers the combination of pinhole and fisheye cameras rather than the full range of heterogeneous sensors. To probe performance beyond these two camera types, we ran a preliminary zero-shot evaluation on 2,014 Matterport3D-360 panoramas:

MethodParamsScale AbsRel ↓AbsRel ↓RMSE ↓δ1\delta_1 ↑τ1.03\tau_{1.03} ↑
DAC0.06B0.15810.20651.95710.64750.1701
UniDAC0.36B0.16630.15921.89330.68510.1789
X-Lens0.04B0.20300.37552.47810.55120.1407

X-Lens trails both baselines here. We attribute this mainly to the training distribution: X-Lens never observes equirectangular inputs, whereas DAC adopts an equirectangular representation as its unified camera space. These results are in Appendix H.4 (Table 20, Figure 16) of the revised PDF, and they support a narrower framing. We have revised Section 5 to describe our evaluation setting as heterogeneous fisheye–pinhole rigs. The Limitations (Appendix G) now state that telephoto, panoramic, and RGB-D sensors are not covered.

(Revised in the paper: Lines 375 and 410 in Section 5, Lines 1611–1615 in Appendix G, and Lines 1883–1938 in Appendix H.4, with new Table 20 and Figure 16.)


W2: Generalization claims

Thank you for this comment. We agree that the main experiments alone do not establish broad generalization, and we will separate two claims more clearly.

  1. Held-out scenes. Training and test splits are strictly separated. OmniScene test scenes are excluded from training and validation, and KITTI360 uses disjoint sequences for training and testing. These experiments measure generalization to unseen scenes within each dataset.
  2. Zero-shot cross-dataset transfer. ETH3D and OmniOcc are real-world datasets not used in training. X-Lens achieves AbsRel 0.0445 and 0.0656 and the lowest Scale AbsRel among the reported methods on both (Table 7).

We have revised the introduction and the contribution list so that zero-shot generalization refers only to the second category.

(Revised in the paper: Lines 99–101 in Section 1 and Lines 130–132 in the contribution list.)

AuthorsOpenReview ↗

Author Response [2/4]

W3: Interaction between task heads

Thank you for this comment. The paper contains one task-interaction ablation: Appendix E.2 (Table 14) adds an auxiliary task branch to the Stage-1 model. That branch consists of camera tokens with a head predicting intrinsics and extrinsics, plus a second DPT head predicting ray maps, trained jointly with depth through a point loss. Data, schedule, and parameter budget are otherwise unchanged.

Variant (six-view OmniOcc, after Stage 1)Scale AbsRel ↓AbsRel ↓RMSE ↓δ1\delta_1 ↑τ1.03\tau_{1.03} ↑
With camera head + ray-map head0.18160.07020.28610.89610.4910
Depth, confidence, and scale heads only (ours)0.06920.06360.24990.94810.5442

Adding the camera and ray-map tasks degrades every depth metric, and Scale AbsRel rises from 0.0692 to 0.1816. For a compact 0.04B model, this suggests that extra geometric tasks compete for capacity rather than regularize the shared features. This motivated our choice to keep only the outputs that serve metric depth directly. The confidence map weights the dense-depth loss and the global scale pooling, and the validity mask excludes sky and out-of-range regions.

We agree that this does not amount to a systematic study. A progressive removal of the remaining heads, with an analysis of their effect on intermediate features, would require retraining several variants, which exceeds our compute budget during the rebuttal period. We now state this remaining gap explicitly in Appendix E.2 and in the Limitations (Appendix G).

(Revised in the paper: Lines 1511–1514 in Appendix E.2 and Lines 1620–1622 in Appendix G.)


Q1: Real-world joint fisheye–pinhole quantitative evaluation

Thank you for this suggestion. As discussed in Appendix G, joint real-world evaluation is limited by the availability of suitable benchmarks. Our own five-camera rig (Figure 6, Appendix D.6) has no LiDAR, so it provides no ground-truth depth. KITTI360 provides both fisheye and pinhole cameras with LiDAR ground truth, although the views overlap very little. We evaluate joint inference with two fisheye views and one pinhole view on KITTI360 sequence 0:

MethodParamsScale AbsRel ↓AbsRel ↓RMSE ↓δ1\delta_1 ↑τ1.03\tau_{1.03} ↑
DAC0.06B0.20250.27784.51240.56410.1137
UniDAC0.36B0.16720.22943.50770.73320.1588
X-Lens0.04B0.17310.23313.92520.69180.1401

On real joint fisheye–pinhole inputs, X-Lens outperforms DAC on all metrics and is close to UniDAC with 89% fewer parameters. UniDAC remains slightly better. This setting is unfavorable to joint inference because the fisheye and pinhole fields of view barely overlap, which leaves little shared evidence for cross-camera reasoning (see Q2). These results are in Appendix H.3 (Table 19), with qualitative comparisons for two scenes in Figure 15 of the revised PDF.

(Revised in the paper: Lines 420–422 in Section 5, Lines 1603–1606 in Appendix G, and Lines 1858–1881 in Appendix H.3, with new Table 19 and Figure 15.)


Q2: Domain gap, sim-to-real, and failure cases

We measure the domain gap through zero-shot performance on real-world data never seen in training, using Scale AbsRel, AbsRel, RMSE, and threshold accuracies. On ETH3D and OmniOcc, X-Lens reaches AbsRel 0.0445 and 0.0656 and the lowest Scale AbsRel among the reported methods (Table 7). Figure 6 shows qualitative results in real fisheye and pinhole scenes. We believe this transfer comes partly from Stage 1, where the shared backbone is trained on large-scale real pinhole data and acquires real-world appearance and geometry priors before heterogeneous training. Appendix D.4 further evaluates robustness to changes in fisheye intrinsics.

The main failure case we observe is very limited overlap between views, which can produce inaccurate scale. The KITTI360 experiment in Q1 illustrates this: with nearly non-overlapping fisheye and pinhole views, X-Lens no longer outperforms the per-view UniDAC baseline. We have added this analysis to Appendix H.3 and the Limitations (Appendix G).

(Revised in the paper: Lines 498–502 in Section 7, Lines 1603–1606 in Appendix G, and Lines 1875–1881 in Appendix H.3, with Table 19 and Figure 15.)

AuthorsOpenReview ↗

Author Response [3/4]

Q3: Single-stage vs. three-stage training

Single-stage training is possible in principle. The three stages separate general geometry learning, camera-specific adaptation, and heterogeneous fusion.

  • Stage 1 uses diverse pinhole datasets to establish depth and metric-scale priors.
  • Stage 2 updates only the calibration tokens and freezes the shared network, adapting to fisheye projection without modifying the pinhole representation.
  • Stage 3 jointly optimizes the network with JDB to learn cross-camera interactions that token-only adaptation cannot capture.

The ablations support this design. Stage 2 reduces fisheye Scale AbsRel from 0.7325 to 0.2126 while leaving pinhole performance unchanged (Table 12). On heterogeneous inputs, joint optimization reduces AbsRel from 0.2333 (frozen backbone and decoder) to 0.1021 (Table 3). Table 12 also shows that joint training without JDB raises pinhole Scale AbsRel on OmniOcc from 0.1052 to 0.5631, which indicates that unstructured heterogeneous training can overwrite previously learned pinhole geometry. A direct single-stage comparison is not yet included, and we list it as a limitation in Appendix G.

(Revised in the paper: Lines 1620–1622 in Appendix G.)


Q4: Why a scale head rather than a constraint on the mean depth

Our factorization follows MapAnything (Keetha et al., 2026), which predicts metric scale separately from up-to-scale geometry. It separates dense shape estimation from global scale prediction. With D^m=m^ D^\hat D^{m}=\hat m\,\hat D, changing m^\hat m rescales all views uniformly and preserves the relative geometry encoded in D^\hat D.

Constraining the mean predicted depth is a reasonable alternative, but local depth errors would then propagate directly into the estimated scale. Our scale head predicts one scale shared by all views from confidence-weighted features, so reliable regions contribute more. This matters for heterogeneous rigs, where distortion and prediction reliability vary across views and image regions. The design also allows scale and relative geometry to be supervised separately. We have added this rationale to Appendix A.1.

(Revised in the paper: Lines 887–893 in Appendix A.1.)


Q5: UniDepthV2 accuracy with the larger model

Thank you for pointing this out. We rechecked our evaluation code and reran both UniDepthV2 variants, obtaining results consistent with the reported values. The change is metric-dependent. On single-view KITTI360 (Table 2), the Large model improves δ1\delta_1 from 0.6931 to 0.7091 while AbsRel and RMSE increase. On OmniScene-Single (Table 6), it improves Scale AbsRel, δ1\delta_1, and τ1.03\tau_{1.03} but has a much larger RMSE. These results are compatible: threshold accuracy counts the fraction of pixels within a tolerance, whereas error metrics, RMSE in particular, are dominated by a small number of large errors.

A plausible contributing factor is the domain shift to fisheye projection. Greater capacity does not necessarily improve robustness to a different camera model, and UniDepthV2 itself discusses limitations with non-pinhole geometry (Piccinelli et al., 2025, Sec. IV-D). The aggregate results alone do not establish the exact cause. We have added this analysis to Appendix D.1.

(Revised in the paper: Lines 1191–1198 in Appendix D.1.)

Reviewer_jhS6OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes X-Lens, a compact feed-forward multi-view metric depth estimation from heterogeneous cameras, i.e., pinhole and fisheye cameras. X-Lens predicts up-to-scale depth maps and metric scale through multi-stage training. The first stage trains on pinhole data and then adapts to fisheye data through calibration tokens in the second stage. The last stage bridges pinhole and fisheye data through the proposed Jacobian Distortion Bias (JDB). Additionally, the paper introduces OmniScene, a synthetic dataset with synchronized pinhole and fisheye multi-view images and metric depth maps.

Critical strengths

-The paper has a clear motivation, problem formulation, and demonstration of the proposed solution. -The multi-stage training helps prevent the model from becoming confused by multiple optimization goals. -The proposed Jacobian Distortion Bias provides a geometric prior to bridge heterogeneous cameras. -The modularity of the proposed solution makes it versatile and can be applied to other works. -OmniScene dataset is a valuable contribution to the community, helping address the scarcity of metric depth datasets for heterogeneous cameras.

Critical weaknesses

-X-Lens has been trained on OmniScene along with KITTI360 and tested on KITTI360. Whereas DAC and UniDAC have not been trained on KITTI360. Therefore, the Table 1 single-view evaluation might not be a fair comparison. Although Table 6 provides a fair comparison, it only compares the multi-view performance, which the monocular models are known to struggle with. A table similar to Table 6 with a comparison in the monocular setting would better highlight the benefit of X-Lens.

-UniDepth, Metric3D, DAC and UniDAC have only been trained on pinhole data whereas X-Lens has been trained on combination of pinhole and fisheye during stage3. While these prior methods offer a good baseline, a stronger baseline would be UniK3D, which has been trained on pinhole, fisheye, and 360° images.

Questions for authors

-How is Scale AbsRel computed for prior monocular methods that don’t explicitly output scale? -Why are the DAC/UniDAC numbers identical in Table 2 (Quad) and Table 6 (Single)?

  • X-Lens is proposed as a heterogeneous camera model, but it has only been evaluated on pinhole and fisheye cameras. How does it perform on panoramic/360° cameras?
3 comments
AuthorsOpenReview ↗

Author Response [1/3]

We sincerely thank the reviewer for the positive assessment and constructive comments. We are glad the reviewer found the motivation clear, the multi-stage training and Jacobian Distortion Bias (JDB) well motivated, the design modular, and OmniScene valuable to the community. Below, we first give a brief overview of our responses, followed by detailed clarifications.

  • Fair monocular comparison. We add a monocular fisheye comparison on OmniScene-Single against DAC and UniDAC fine-tuned on OmniScene. X-Lens achieves the best AbsRel, RMSE, δ1\delta_1, and τ1.03\tau_{1.03}, while fine-tuned UniDAC attains a lower Scale AbsRel.
  • UniK3D baseline. We evaluate UniK3D-Small/Large on all benchmarks and add them to the revised appendix (Appendix H.1, Table 17). X-Lens is better on every metric on the heterogeneous OmniScene-Full and OmniScene-Quad settings. UniK3D-Large is stronger on several dense-depth metrics on the pinhole benchmarks, at 9× the parameters and lower throughput.
  • Scale AbsRel. It is computed from the final metric depth of every method and does not require a scale head. The identical DAC/UniDAC numbers follow from both tables using the same fisheye images with per-view inference.
  • Panoramic cameras. We add a preliminary zero-shot study on Matterport3D-360. X-Lens, which never sees panoramic input during training, trails DAC and UniDAC there. We will state this scope limitation explicitly.

W1: Fairness of the single-view comparison

Thank you for this suggestion. We agree that the KITTI360 single-view results (Table 2) mix two settings: X-Lens uses the KITTI360 training split, while DAC and UniDAC are evaluated zero-shot. We believe the fair comparison the reviewer refers to is Table 9, which reports DAC and UniDAC fine-tuned on OmniScene under the six-view OmniScene-Full protocol. Following your suggestion, we also evaluate the same fine-tuned checkpoints in the monocular OmniScene-Single setting (one fisheye view, 504 × 798):

MethodParamsScale AbsRel ↓AbsRel ↓RMSE ↓δ1\delta_1 ↑τ1.03\tau_{1.03} ↑
DAC0.06B0.26810.19632.89110.79500.2771
DAC + FT0.06B0.23810.16712.27510.81500.2969
UniDAC0.36B0.27190.18692.89030.76930.2691
UniDAC + FT0.36B0.21530.12812.13490.82910.3077
X-Lens0.04B0.22870.11911.98010.88680.3357

With the same target-domain supervision, X-Lens achieves the best dense-depth error and accuracy metrics using 89% fewer parameters than UniDAC. Fine-tuned UniDAC obtains a lower Scale AbsRel, so the monocular advantage is not uniform across metrics. Because DAC and UniDAC process each image independently, their per-view predictions are the same here as in the four-view OmniScene-Quad setting. X-Lens, in contrast, improves from 0.2287 to 0.1268 Scale AbsRel when moving from one to four views (Table 2). This gap reflects the contribution of joint multi-view inference. We have added this table to the revised appendix (Appendix H.2, Table 18). Appendix C.4 states the training-data difference in the KITTI360 comparison.

(Revised in the paper: Lines 1836–1856, Appendix H.2, with new Table 18.)

AuthorsOpenReview ↗

Author Response [2/3]

W2: UniK3D as a stronger baseline

Thank you for this suggestion. We evaluated UniK3D (Small and Large) under the same protocols and added the results to the revised appendix (Appendix H.1, Table 17). UniK3D is applied independently to each view, like DAC and UniDAC. X-Lens results are reproduced from the paper.

BenchmarkMethodParamsScale AbsRel ↓AbsRel ↓RMSE ↓δ1\delta_1 ↑τ1.03\tau_{1.03} ↑FPS ↑
OmniScene-Full (6 views)UniK3D-Small0.04B0.40320.14742.85610.82230.24825
UniK3D-Large0.36B0.20560.12682.42060.86680.34912
X-Lens0.04B0.11810.10211.59930.89820.372422
KITTI360 (1 view)UniK3D-Small0.04B0.11180.24066.41770.68730.143217
UniK3D-Large0.36B0.10340.22246.27710.76840.19386
X-Lens0.04B0.09550.21104.19610.69590.143641
KITTI360 (2 views)UniK3D-Small0.04B0.11180.24066.41770.68730.14328
UniK3D-Large0.36B0.10340.22246.27710.76840.19383
X-Lens0.04B0.08320.20694.04330.70140.155741
OmniScene-Quad (4 views)UniK3D-Small0.04B0.38710.16092.71530.79530.18807
UniK3D-Large0.36B0.20620.11712.58610.86920.29733
X-Lens0.04B0.12680.11381.64810.88220.332824
OmniScene-Single (1 view)UniK3D-Small0.04B0.38710.16092.71530.79530.188028
UniK3D-Large0.36B0.20620.11712.58610.86920.297312
X-Lens0.04B0.22870.11911.98010.88680.335734
ETH3D (2 views)UniK3D-Small0.04B0.16560.03450.61650.98360.649523
UniK3D-Large0.36B0.10020.01850.38990.99670.84748
X-Lens0.04B0.12170.04450.69910.97230.555239
ScanNet++V2 (2 views)UniK3D-Small0.04B0.20880.07390.17990.94660.390323
UniK3D-Large0.36B0.12980.04160.13700.97370.66268
X-Lens0.04B0.05440.05490.16210.96260.542339
OmniOcc (6 views)UniK3D-Small0.04B0.12940.06390.25090.95860.48598
UniK3D-Large0.36B0.16040.05560.22280.96570.60463
X-Lens0.04B0.06700.06560.25840.94940.555026

At an equal parameter budget (0.04B), X-Lens outperforms UniK3D-Small on all metrics in every fisheye and heterogeneous setting and has a lower Scale AbsRel on every benchmark, although UniK3D-Small obtains better dense-depth metrics on ETH3D and OmniOcc. In the heterogeneous OmniScene-Full and OmniScene-Quad settings, X-Lens also outperforms UniK3D-Large on all metrics, reducing Scale AbsRel by 43% and 39%, respectively. On KITTI360, X-Lens obtains lower Scale AbsRel, AbsRel, and RMSE, while UniK3D-Large has higher threshold accuracies. On the pinhole benchmarks, UniK3D-Large is stronger on most dense-depth metrics. X-Lens retains a lower Scale AbsRel on ScanNet++V2 and OmniOcc with 9× fewer parameters and 3–13× higher throughput. Appendix H.1 states these trade-offs explicitly.

(Revised in the paper: Lines 420–422 in Section 5, Lines 1150–1152 in Appendix C.4, and Lines 1789–1831 in Appendix H.1, with new Table 17. A qualifying note is added in Lines 1187–1189 of Appendix D.1.)

AuthorsOpenReview ↗

Author Response [3/3]

Q1: Computing Scale AbsRel for methods without a scale head, and identical DAC/UniDAC numbers

Thank you for requesting this clarification. Scale AbsRel is computed from the final metric depth predictions and does not require an explicit scale head. For each evaluation sample, we compute the mean predicted depth μpred\mu_{\mathrm{pred}} and the mean ground-truth depth μGT\mu_{\mathrm{GT}} over the same valid pixels, and report

Scale AbsRel=∣μpred−μGT∣μGT.\mathrm{Scale\ AbsRel}=\frac{|\mu_{\mathrm{pred}}-\mu_{\mathrm{GT}}|}{\mu_{\mathrm{GT}}}.

For multi-view samples, valid pixels are pooled across the evaluated views before computing these means, and the resulting errors are averaged across samples. Any method that outputs metric depth, including monocular ones, can therefore be evaluated with the same definition. No ground-truth scale alignment is applied when computing Scale AbsRel; alignment is used only for the separate relative-depth metrics.

DAC and UniDAC process each image independently, without cross-view fusion. OmniScene-Single (Table 6) evaluates exactly the same fisheye images as OmniScene-Quad (Table 2). Because the baselines run per view in Quad, both tables report the same monocular predictions. X-Lens instead processes all four views jointly in Quad, so its results differ from its single-view setting.

We have added the full definition to Appendix C.4 and a pointer in Section 5:

Scale AbsRel is computed from each method's final metric depth as ∣μpred−μGT∣/μGT|\mu_{\mathrm{pred}}-\mu_{\mathrm{GT}}|/\mu_{\mathrm{GT}}, where both means are taken over the valid pixels of all evaluated views in a sample, without ground-truth scale alignment. Monocular baselines are run independently on each view; their OmniScene-Single and OmniScene-Quad results therefore coincide.

(Revised in the paper: Lines 415–416 in Section 5 and Lines 1160–1171 in Appendix C.4.)


Q2: Panoramic / 360° cameras

We acknowledge this limitation. OmniScene contains only fisheye and pinhole images, so our training and main evaluations focus on these two camera types. We have not systematically validated X-Lens on panoramic cameras. To probe this, we ran a preliminary zero-shot evaluation on 2,014 Matterport3D-360 panoramas:

MethodParamsScale AbsRel ↓AbsRel ↓RMSE ↓δ1\delta_1 ↑τ1.03\tau_{1.03} ↑
DAC0.06B0.15810.20651.95710.64750.1701
UniDAC0.36B0.16630.15921.89330.68510.1789
X-Lens0.04B0.20300.37552.47810.55120.1407

X-Lens trails both baselines on this benchmark. We attribute this mainly to the training distribution: X-Lens never observes equirectangular inputs, whereas DAC adopts an equirectangular representation as its unified camera space. Qualitative comparisons are shown in Figure 16 (Appendix H.4) of the revised PDF. We now state this scope explicitly in Section 5 and in the Limitations (Appendix G):

X-Lens is trained and evaluated on fisheye and pinhole cameras. Its zero-shot transfer to equirectangular panoramas is limited (Appendix H.4), and extending training to panoramic and other projection models is left for future work.

(Revised in the paper: Lines 375 and 410 in Section 5, Lines 1611–1615 in Appendix G, and Lines 1883–1938 in Appendix H.4, with new Table 20 and Figure 16.)

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their time and effort in handling our submission, and for the detailed and constructive feedback, which has improved the paper. We appreciate that the reviewers recognized the clear motivation, the well-motivated multi-stage training and JDB, and the modular design (jhS6), the ray-space formulation and the separation of lens adaptation from cross-view interaction (dqa9), the compact 0.04B model with edge deployment (CNtA), and OmniScene as a valuable dataset (jhS6, CNtA).

As the rebuttal period nears its end, we summarize the rebuttal content below. All revisions are marked in orange in the PDF.

  1. Real-world joint fisheye–pinhole evaluation (CNtA W2/Q1; dqa9 W1). We added a quantitative evaluation on real KITTI360 with two fisheye views, one pinhole view, and LiDAR GT (CNtA [2/4] Q1; dqa9 [1/2] W1; App. H.3, Tab. 19, Fig. 15). X-Lens outperforms DAC on all metrics and approaches UniDAC with 89% fewer parameters, although UniDAC remains slightly better because the views barely overlap. CNtA still considers this evidence insufficient. We agree that a real heterogeneous benchmark with dense GT and sufficient overlap is scarce, and we state this in App. G and the Conclusion.

  2. Generalization claims (CNtA W2/Q2). We separate held-out generalization from zero-shot transfer (CNtA [1/4] W2, [2/4] Q2; Sec. 1, L99–101, L130–132). Held-out generalization covers disjoint OmniScene and KITTI360 splits and unseen intrinsics (Tab. 1, 2, 6, 8). For zero-shot transfer to real ETH3D and OmniOcc, X-Lens achieves the lowest Scale AbsRel (Tab. 7). Limited view overlap is reported as the main failure case (App. G, H.3). CNtA welcomed the clarified wording but still requests more evidence.

  3. Scope and panoramic inputs (jhS6 Q; CNtA W1; dqa9 W5). In a zero-shot Matterport3D-360 study, X-Lens trails DAC and UniDAC because it never sees equirectangular inputs (jhS6 [3/3] Q2; CNtA [1/4] W1; dqa9 [2/2] W5; App. H.4, Tab. 20, Fig. 16). We therefore narrowed the scope to fisheye–pinhole rigs (Sec. 5, App. G). Panorama-based methods operate on stitched inputs and address a different setting.

  4. Baselines and fairness (jhS6 W1/W2; CNtA Q6). Against fine-tuned DAC and UniDAC on monocular OmniScene-Single, X-Lens is best on AbsRel, RMSE, δ1 and τ, while fine-tuned UniDAC has a lower Scale AbsRel (jhS6 [1/3] W1; App. H.2, Tab. 18). Against UniK3D, X-Lens is better on all metrics on OmniScene-Full and OmniScene-Quad. On pinhole benchmarks, UniK3D-Large is stronger on most dense metrics, at 9× the parameters and 3–13× lower speed (jhS6 [2/3] W2; App. H.1, Tab. 17). Main tables use the original DAC checkpoint, and fine-tuned variants are labeled “+FT” (CNtA [4/4] Q6; App. D.5).

  5. Design choices (CNtA W3/Q3/Q4). The ablations support the three training stages (Tab. 3, 12) and show that additional camera and ray-map heads degrade depth (Tab. 14). Following MapAnything, the scale head pools confidence-weighted features across views (CNtA [2/4] W3, [3/4] Q3/Q4; App. A.1, E.2). A direct single-stage baseline and a systematic head-removal study remain open, and we list both as limitations (App. G).

  6. JDB, calibration, and global scale (dqa9 W2–W4, Q1/Q2). JDB is a soft geometric prior, not an epipolar constraint (Sec. 3.2). X-Lens assumes calibrated rigs, since predicting calibration jointly hurt depth (Tab. 14), and App. D.4 evaluates robustness to perturbed intrinsics. Confidence-weighted pooling cannot recover missing cross-view evidence. We acknowledge these points as limitations (dqa9 [1/2], [2/2]; App. G, H.3).

  7. Evaluation clarifications (jhS6 Q1; CNtA Q5/Q7). Scale AbsRel is computed from the final metric depth without GT alignment, following MapAnything, so it also applies to methods without a scale head. The identical DAC and UniDAC numbers in OmniScene-Single and OmniScene-Quad follow from per-view inference. We re-verified the UniDepthV2 trend and found it metric-dependent (jhS6 [3/3] Q1; CNtA [3/4] Q5, [4/4] Q7; Sec. 5, App. C.4, D.1).

In summary, the rebuttal adds new quantitative results and narrows the claims to fisheye–pinhole rigs. It also states the remaining limitations explicitly. Reviewer dqa9 maintained a positive score. Reviewer CNtA kept the original assessment, mainly regarding real-world heterogeneous benchmarks and the remaining design ablations, which were provided in our original rebuttal.

We guarantee the public release of the code and the OmniScene dataset, and welcome any further inquiries.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.