acceptodds
Under review as a conference paper at ICLR 2027

X3Calib: Single Image LiDAR-Camera Calibration as Cross-Modal 3D Registration

Abstract

Extrinsic calibration between a LiDAR and a camera is a prerequisite for sensor fusion, yet learned calibrators must match 3D geometry against 2D appearance, and most are validated only on the dataset they were trained on. We reformulate calibration from a single LiDAR scan and a single image as 3D-3D registration. A monocular metric depth network lifts the image into a metric point cloud, so the LiDAR scan and the image become two point sets in one geometric domain. An overlap-aware correspondence network, trained only on KITTI, matches the two clouds, and iterated least squares with a coarse-to-fine inlier threshold turns the matches into extrinsics, with either a PnP or a 3D-3D solver. Zero-shot on nuScenes, Argoverse, and PandaSet, X3Calib outperforms LCCNet and CalibNet, neither of which even reaches the recall of simply returning the identity transform. On nuScenes, we also evaluate the recent CMRNext, which was trained on the other three datasets: X3Calib exceeds its recall by a wide margin, and CMRNext's recall again falls below that of the identity transform. A simple pipeline built on 3D registration thus generalizes better than dedicated calibration networks, even one trained on more data. The code will be made public upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LiDAR-camera calibration, Cross-modal registration, Monocular metric depth, 3D-3D correspondence learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WZ8UOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce X3Calib, a single-frame LiDAR-camera (cross-modal) calibration method. By using a monocular depth estimation network, EasyCalib can then lift the 2D features to 3D to then find 3D geometric correspondences between the 2 modalities using a graph neural network (GNN). These geometric correspondences are then provided to a classic solver, such as PnP, to arrive at the final extrinsic parameters (rotation, translation), making X3Calib a hybrid cross-modal calibration method. The authors train and evaluate X3Calib on multiple datasets to evaluate against heterogeneous sensors and different extrinsics. The authors find that

Critical strengths

The authors correctly identifies that common datasets such as KITTI and NuScenes each only offer 1 extrinsic because of the established sensor suite used for data collection. Accordingly, the authors evaluate on multiple datasets to understand how X3Calib performs on multiple extrinsics.

Given that this calibration requires finding the spatial relationship (6 DoF relative transform) between two sensors, the idea to find the correspondences in 3D space is well motivated instead of between 2D and 3D features.

Critical weaknesses

There are newer cross-modal calibration baselines beyond LCCNet and CalibNet ([1][2][3]). Evaluating on some or all of these baselines would be helpful in understanding X3Calib's competitiveness. Furthermore, two of these newer baselines, BEVCalib and CalibBEV, follow a similar paradigm like X3Calib in performing the cross-modal calibration in the BEV grid.

The Identity baseline is unnecessary. If the extrinsic is not an identity transform, then the reported error is not meaningful because it's clearly wrong. The CalibNet baseline also reports translation and rotation error on roughly the same scale.

[1] Yuan, W., Li, J., Yue, J., Shah, D., Karydis, K., & Qiu, H. (2025). BEVCalib: LiDAR-camera calibration via geometry-guided bird’s-eye view representation. Proceedings of Machine Learning Research, 305, 4824–4836. https://proceedings.mlr.press/v305/yuan25a.html

[2] D’Addeo, F., Cipelli, L., Cardace, A., Ghelfi, E., Zinelli, A., & Bertozzi, M. (2026). CalibBEV: LiDAR-camera calibration via BEV alignment. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 4345–4354). https://doi.org/10.1109/WACV61042.2026.00423

[3] Ou, N., Chen, Z., Zhang, X., & Wang, J. (2026). Native-domain cross-attention for camera–LiDAR extrinsic calibration under large initial perturbations. IEEE Robotics and Automation Letters. https://doi.org/10.1109/LRA.2026.3683597

Questions for authors

Evaluate with more recent baselines that are also singe-frame calibration methods.

May the authors identify X3Calib's failure cases? Providing axis-wise errors for both rotation and translation error could help in determining this.

Minor nit: For the qualitative figures (Fig 2 and 4), change the depth scale to make closer objects more distinguishable.

1 comment

We thank Reviewer WZ8U for the review and the pointers to recent methods. We are glad that the reviewer recognizes the need to evaluate across several sensor rigs, since KITTI and nuScenes each offer one extrinsic, and finds matching in 3D, rather than between 2D and 3D features, well motivated. The global response explains the updated schedule and the new baselines (Tables R1–R3). Triples give recall (%) at (1°, 0.25 m) / (5°, 0.5 m) / (10°, 1 m).

W1, Q1. How does X3Calib compare with newer single-frame methods? Two of them, BEVCalib [1] and CalibBEV [2], calibrate in a BEV grid, like X3Calib.

First, a clarification: X3Calib uses no BEV grid. It registers the LiDAR scan with the depth-lifted image cloud directly in 3D and solves for the pose from explicit correspondences, which, as Reviewer ekLy notes, “improve interpretability and facilitate subsequent error analysis and modular improvements.”

We ran BEVCalib [1], which releases code and trained weights, on all four datasets with its released preprocessing; CalibBEV [2] releases neither. We also added PseudoCal, the closest prior work, trained by us (global response).

  • KITTI (Table R1): BEVCalib's weights saw our test sequences. Among held-out methods, X3Calib leads at (1°, 0.25 m), with 76.0% against PseudoCal's 61.2%, and in rotation (median 0.41° against 0.78°).
  • Zero-shot: BEVCalib does not estimate translation from the data. On KITTI, a mount-shift test shows that it returns KITTI's LiDAR mounting position, and zero-shot its translation is off by 2.9–4.6 m. Rotation is therefore the like-for-like comparison (Table R3): on Argoverse and PandaSet, X3Calib has 41.1% and 32.0% of frames under 1°, against at most 1.9% and 0.9% for BEVCalib and PseudoCal (p < 10⁻⁴⁵). On nuScenes, all methods fall below no correction.
  • With 10 frames of one drive, X3Calib's rotation lies under 1° in 89% of Argoverse and 47% of PandaSet groups, against 0% for BEVCalib.

W2. Is the Identity baseline necessary? If the true extrinsic is not the identity, its error seems meaningless, and CalibNet's errors are on the same scale.

We agree that the name misleads, and we rename the baseline No correction (initial extrinsic). It applies the identity correction, not the identity extrinsic: it returns the perturbed initial extrinsic unchanged, so its error equals the perturbation. Because the perturbation is bounded, this floor already scores 13–16% at (5°, 0.5 m) and 47–52% at (10°, 1 m) (Tables R1, R2); a method below it makes calibration worse than not running it. That CalibNet's errors lie on the same scale is what the row reveals: zero-shot, CalibNet falls below the floor on all three unseen datasets (Tab. 2 of the paper), as do BEVCalib and PseudoCal (Table R2).

Q2. What are X3Calib's failure cases? Per-axis errors would help to identify them.

  • Per axis (Table R8): zero-shot, the largest rotation error is about the yaw axis, and the largest translation error lies along the optical axis, which a single forward view constrains most weakly. On KITTI, every axis stays within 0.2° and 0.11 m.
  • Failure cases (Table R9): the main failure is nuScenes, whose 32-beam scans leave a median of 3 ground-truth correspondences per frame, against 40 on Argoverse and 47 on PandaSet. Elsewhere, failures concentrate in frames with few correct matches, and the number of matches that agree with the final pose flags them without ground truth. The revision shows successes and both failure types on images, chosen by a fixed rule.

Table R8. Per-axis median error of X3Calib, rotation (°) and translation (m; z = optical axis)

DatasetPitchYawRollxyz
KITTI0.150.200.190.060.040.11
nuScenes2.306.773.600.310.260.36
Argoverse0.240.850.510.240.130.43
PandaSet0.341.650.690.280.200.44

No correction: 3.4–4.0° and 0.24–0.26 m per axis.

Table R9. Zero-shot recall at (5°, 0.5 m) by correct matches per frame (uses ground truth), for the 25% of frames with the most matches agreeing with the final pose (no ground truth), and overall

Dataset01–45–910–1920–49Top 25%All
nuScenes5.511.025.0––24.77.8
Argoverse13.016.830.247.358.659.336.0
PandaSet9.711.333.054.484.874.428.9

–: at most one frame.

Minor. Could the depth colour scale in Figs. 2 and 4 make close objects easier to distinguish?

Agreed. The revised figures colour points by inverse depth, which spreads out the near range.

Reviewer_ekLyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes X3Calib, a single-frame LiDAR-camera extrinsic calibration method based on cross-modal 3D point cloud registration. The method converts a monocular image into a 3D point cloud using metric depth estimation, establishes cross-modal point cloud correspondences through the PREDATOR network, and estimates the extrinsic parameters via iterative least-squares optimization. The method is trained on KITTI and evaluated in a zero-shot setting on nuScenes, Argoverse, and PandaSet. Experimental results demonstrate that X3Calib achieves better cross-dataset generalization than existing learning-based calibration methods.

Critical strengths

This paper reformulates the cross-modal LiDAR-camera calibration problem as a 3D point cloud registration task, offering a simple and practically valuable solution.

By explicitly establishing cross-modal point cloud correspondences rather than directly regressing the extrinsic parameters, the proposed method improves interpretability and facilitates subsequent error analysis and modular improvements.

The paper evaluates zero-shot generalization on three datasets with different sensor configurations and explicitly distinguishes baseline results affected by training-data overlap, making the experimental evaluation more comprehensive and rigorous.

The paper introduces an Identity baseline to assess whether different methods genuinely improve upon the initial calibration, making the cross-dataset performance comparisons more convincing.

Critical weaknesses
  1. The median translation errors of X3Calib on the three unseen datasets are nearly identical to those of the Identity baseline, indicating that its cross-domain generalization advantage primarily lies in rotation estimation. Its ability to perform full 6-DoF extrinsic calibration has therefore not been sufficiently demonstrated. 2.The performance improvements of X3Calib in cross-domain evaluations are mainly observed under relatively relaxed error thresholds, while its recall remains very low under stricter thresholds. These results are insufficient to demonstrate that the proposed method can achieve high-precision cross-domain LiDAR-camera calibration.
  2. The existing ablation studies mainly focus on the final pose estimation strategies, lacking systematic evaluations of metric depth estimation, cross-modal correspondence learning, and key module designs. Consequently, it is difficult to determine the actual contributions of the core components to the observed performance improvements. 4.Although the paper identifies PseudoCal as the most closely related prior work, it lacks a quantitative comparison with PseudoCal under a unified experimental setting. As a result, the performance advantages of the proposed method over the most closely related approach have not been sufficiently validated.
Questions for authors

1.The authors acknowledge that the publicly released weights of LCCNet and CMRNext were trained on data that include the KITTI test sequences used in this paper. Why were these baseline methods not retrained using strictly separated training and test splits to ensure a fair in-distribution performance comparison?

2.The median translation errors of X3Calib on the three unseen datasets are nearly identical to those of the Identity baseline. Could the authors further explain the reasons for the limited improvement in cross-domain translation estimation and provide additional evidence to validate the method's generalization capability for full 6-DoF extrinsic calibration?

3.X3Calib achieves relatively low recall under strict error thresholds in cross-domain evaluations, and even on KITTI, its Recall@(1°, 0.25m) is only 57.7%. Could the authors further analyze the main factors limiting the method's high-precision calibration performance?

4.The current ablation studies mainly focus on pose estimation strategies. Could the authors provide additional ablation experiments on different depth estimation models, PREDATOR pretraining and fine-tuning strategies, and key matching components to better assess the contribution of each component to the overall performance?

5.PseudoCal adopts a similar calibration approach based on monocular depth-derived pseudo point clouds. Could the authors provide a quantitative comparison with PseudoCal to more thoroughly validate the relative advantages of X3Calib?

1 comment

We thank Reviewer ekLy for a thorough review. We are glad that the reviewer finds the registration formulation simple and practically valuable, its explicit correspondences interpretable, and the zero-shot evaluation on three rigs, with its Identity baseline, rigorous. The global response explains the updated schedule and the new baselines (Tables R1–R3). Triples give recall (%) at (1°, 0.25 m) / (5°, 0.5 m) / (10°, 1 m).

W1, Q2. Zero-shot median translation errors nearly equal Identity's, so 6-DoF calibration is not shown. What limits translation, and what supports 6-DoF generalization?

A single forward view constrains translation most weakly along the optical axis, and the 3D-3D residual of the submitted schedule turns the scale error of the lifted cloud (±1–2%) into exactly that error (Spearman 0.91 on KITTI). The updated schedule refines the pose with the scale-free PnP residual, and X3Calib now estimates all six degrees of freedom zero-shot:

  • Single frames (Table R2): recall at (1°, 0.25 m) / (5°, 0.5 m) reaches 9.7 / 36.0 on Argoverse and 9.1 / 28.9 on PandaSet, against 0.4 / 15.5 and 0.2 / 13.6 without correction.
  • Several frames (new): 10 frames of one Argoverse drive reach 31.1 / 93.3 / 100, with a median translation error of 0.29 m, below the initial 0.48 m.

From one unseen image, the median translation error (0.60–0.68 m) still exceeds the initial 0.48–0.50 m, and nuScenes fails; the revision states both limits.

W2, Q3. Recall at strict thresholds is low, and only 57.7% at (1°, 0.25 m) on KITTI. What limits high-precision calibration?

  • Depth scale. With the true per-frame scale as an oracle, the submitted solve reaches 78.5% on KITTI instead of 57.7%. The updated schedule, whose last rounds use the scale-free PnP residual, reaches 76.0% (Table R1). Rotation is already precise: 94.0% of frames lie under 1° (Table R3).
  • Correct matches zero-shot. On PandaSet, recall at (1°, 0.25 m) grows from 1.6% with 1–4 correct matches to 51.5% with 20–49, so matching, not the solver, bounds zero-shot precision. Matches that agree with the final pose flag precise estimates without ground truth: the most-agreeing quarter of frames reaches 59.3% / 74.4% at (5°, 0.5 m) on Argoverse / PandaSet, against 36.0% / 28.9% overall.

W3, Q4. How do the depth model, PREDATOR pretraining and fine-tuning, and the matching components contribute?

  • Depth model (Table R6). On KITTI, depth quality matters: Metric3Dv2 reaches 77% at (1°, 0.25 m), MoGe-2 32%. Zero-shot, 6-DoF calibration needs accurate metric depth: Depth Pro falls below no correction on both datasets.
  • Pretraining and fine-tuning. Frozen PREDATOR, from KITTI LiDAR–LiDAR or 3DMatch weights, reaches at most 0.5% at (5°, 0.5 m) on KITTI, against 97.1% after fine-tuning (Table R5, response to Reviewer HQV4).
  • Matching (Table R7). The mutual check and overlap ranking matter; the matchability score adds nothing beyond overlap.

Table R6. Depth models (100 frames each): KITTI at (1°, 0.25 m), zero-shot at (5°, 0.5 m)

Depth modelKITTIArgoversePandaSet
No correction01711
Metric3Dv2 ViT-L (ours)773029
Metric3Dv2 ViT-g644226
UniDepth v2741117
MoGe-2323436
Depth Anything V2362521
Depth Pro4148

Table R7. Matching: recall at (1°, 0.25 m) / (5°, 0.5 m)

VariantKITTIArgoversePandaSet
X3Calib76.0/97.19.7/36.09.1/28.9
No mutual check66.3/94.96.1/30.73.0/15.3
Similarity ranking75.8/97.08.1/31.14.7/17.2
No matchability76.0/97.110.0/37.38.6/29.8

W4, Q5. How does X3Calib compare quantitatively with PseudoCal?

PseudoCal (Cocheteux et al., BMVC 2023) is the most relevant prior work but releases no trained weights. We therefore trained it, as fairly as possible, with the authors' code on our KITTI training split, and evaluated it on the same frames and perturbations as X3Calib. X3Calib outperforms it at the strict threshold and zero-shot:

  • KITTI (Table R1): 76.0% against 61.2% at (1°, 0.25 m), and a median rotation error of 0.41° against 0.78°. The looser thresholds tie, and PseudoCal's median translation error is lower (0.07 m against 0.16 m).
  • Zero-shot (Table R2): PseudoCal falls below no correction on all three datasets; X3Calib exceeds it on Argoverse and PandaSet.

Its authors call the code a reimplementation rather than a reproduction, so our numbers may understate it.

Q1. Why were LCCNet and CMRNext not retrained on strictly separated splits?

To give each baseline a fair chance, we use public code and weights, run as the authors released them; PseudoCal, which releases no weights, is the one exception. Because the public LCCNet and CMRNext weights saw our KITTI test sequences, their rows stay gray and unranked. PseudoCal, which we trained on our KITTI training split, gives a held-out in-distribution comparison (Table R1). Retraining the baselines on separated splits is future work.

Reviewer_HQV4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes X3Calib for single-frame LiDAR-camera calibration. It converts the image into a metric point cloud, fine-tunes PREDATOR to establish correspondences, and estimates the extrinsics through iterated least squares. Experiments on KITTI and three other driving datasets show cross-dataset gains mainly in rotation estimation.

Critical strengths

The method shows clear cross-dataset rotation improvements on Argoverse and PandaSet over the Identity baseline. 相比Identity基线,该方法在Argoverse和PandaSet上显示了明显的跨数据集旋转估计提升。

The solver comparisons use the same correspondences, providing useful analysis of the effects of depth information and fitting strategy.

Critical weaknesses
  • The technical contribution is limited. PseudoCal already uses depth-lifted point clouds, while X3Calib mainly combines PREDATOR with standard geometric fitting. Comparisons with recent depth-based matching methods such as UniCalib are also missing.
  • The baselines are outdated. Comparisons with more recent methods (e.g., UniCalib and CalibBEV [WACV 2026]) are suggested . Cross-dataset calibration accuracy remains limited, particularly in translation. The median translation errors remain essentially unchanged from Identity, and recall at (1°, 0.25 m) is only 0–2.5% across the three test datasets.
Questions for authors
  • How much does KITTI fine-tuning improve over frozen PREDATOR under the same depth model and solver?
  • What causes the unchanged cross-dataset translation errors? How does the method perform under translation-only perturbations?
  • Are LCCNet’s mean errors and recall computed on the same frames, with the same failure handling?
  • Which Metric3Dv2 checkpoint is used? Its published training data include PandaSet, so the claim of “sensor rigs it has never seen” needs clarification.
1 comment

We thank Reviewer HQV4 for a careful review and precise questions. We are glad that the reviewer finds the paper easy to follow, the rotation gains on Argoverse and PandaSet clear, and the solver comparison on identical correspondences useful. The global response explains the updated schedule and the new baselines (Tables R1–R3). Triples give recall (%) at (1°, 0.25 m) / (5°, 0.5 m) / (10°, 1 m).

W1. Is the contribution limited? PseudoCal already lifts the image with monocular depth, and X3Calib mainly combines PREDATOR with standard fitting.

The contribution is the reformulation, not a new network: PseudoCal regresses the extrinsics, while X3Calib learns cross-modal correspondences and solves for the pose. As Reviewer ekLy notes, this is “a simple and practically valuable solution” whose explicit correspondences improve interpretability and error analysis. Three results show what the correspondences add:

  • PseudoCal, trained by us with its authors' code, reaches 61.2% at (1°, 0.25 m) on KITTI, against 76.0%, and falls below no correction on every unseen dataset (Tables R1, R2).
  • A pose regressor on X3Calib's own frozen features reaches 11.4% on KITTI, against 76.0%, and zero-shot median rotation errors of 7.09° and 5.44° on Argoverse and PandaSet, against 1.26° and 2.26°.
  • Correspondences pool evidence across frames: with 10 frames of one drive, X3Calib's translation error on Argoverse falls to 0.29 m, below the initial 0.48 m.

W1, W2. Are the baselines outdated? How does X3Calib compare with recent methods such as UniCalib and CalibBEV?

We added two methods (global response): BEVCalib (Yuan et al., CoRL 2025), run with its released code and weights, and PseudoCal (Cocheteux et al., BMVC 2023), trained by us. UniCalib (Han et al., arXiv 2025) and CalibBEV (D'Addeo et al., WACV 2026) release neither code nor weights. BEVCalib returns KITTI's mounting position as its translation, so rotation is the like-for-like comparison (Table R3). On KITTI, Argoverse, and PandaSet, X3Calib has the most precise rotation of all held-out methods: 94.0% of frames under 1° on KITTI (PseudoCal 61.2%), and 41.1% and 32.0% on Argoverse and PandaSet (baselines at most 1.9%).

W2, Q2. Zero-shot median translation errors equal Identity's, and recall at (1°, 0.25 m) is 0–2.5%. Why? How does X3Calib handle translation-only perturbations?

With the updated schedule, X3Calib estimates all six degrees of freedom zero-shot: recall at (1°, 0.25 m) reaches 9.7% on Argoverse and 9.1% on PandaSet, against 0.4% and 0.2% without correction (Table R2). The update targets two properties of the submitted schedule: early rounds that accept matches within 30 m (500 px), which let wrong matches pull the translation away, and a 3D-3D residual that turns the scale error of the lifted cloud into translation along the optical axis.

Under translation-only perturbations (Table R4), X3Calib corrects the offset on KITTI, from 0.48 m to 0.16 m median error, but not from one unseen image, a limit the revision states.

Table R4. Translation-only perturbations (R = I, t up to 0.5 m per axis): median translation error (m)

MethodnuScenesArgoversePandaSet
No correction0.490.490.49
X3Calib, updated0.590.560.70

Q1. How much does KITTI fine-tuning improve over frozen PREDATOR with the same depth model and solver?

Without fine-tuning, matching fails (Table R5). Frozen PREDATOR, from its KITTI LiDAR–LiDAR weights (our initialization, which the paper misnames as 3DMatch) or from its 3DMatch weights, falls below no correction on every dataset.

Table R5. Fine-tuning (same depth model and schedule)

WeightsKITTIArgoversePandaSet
No correction0.4/13.6/51.60.4/15.5/51.30.2/13.6/46.7
Frozen, KITTI LiDAR0.0/0.5/2.80.0/0.2/9.10.0/0.6/10.3
Frozen, 3DMatch0.0/0.3/2.50.0/1.1/6.60.0/4.1/23.0
Fine-tuned (X3Calib)76.0/97.1/98.89.7/36.0/70.89.1/28.9/52.2

Q3. Are LCCNet's errors and recall computed on the same frames, with the same failure handling?

All results share one frame list per dataset, one perturbation per frame, one error definition, and one failure rule: a failed estimate counts as no correction.

Q4. Which Metric3Dv2 checkpoint is used? Its training data include PandaSet, so “sensor rigs it has never seen” needs clarification.

We use Metric3Dv2 ViT-L, whose training data include Argoverse2 and PandaSet but not KITTI or nuScenes. We revise the claim: the correspondence network has never seen these rigs, but the depth network may have. Depth Anything V2 (metric outdoor), which saw none of the four datasets, still lifts zero-shot recall at (5°, 0.5 m) from 17% to 25% on Argoverse and from 11% to 21% on PandaSet (100 frames each).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.