acceptodds
Under review as a conference paper at ICLR 2027

CalibBench: Diverse Multimodal Calibration Benchmark

Abstract

Cross-modal sensor fusion is integral to modern autonomous systems for perceiving and interacting with their environments. The safe and successful deployment of these systems depends on ensuring that the distinct sensor modalities are aligned, necessitating cross-modal calibration to identify the extrinsic parameters among the sensors. Current methods seek to address the need for cross-modal calibration in various ways, including target-based and target-less calibration, as well as deep learning. However, existing methods have been evaluated on homogeneous or closed-source datasets, which makes it difficult to develop and validate follow-on methodologies. This work focuses on developing a cross-modal (camera and LiDAR) calibration dataset. We elaborate on how we make our dataset diverse and challenging, and report the performance of previous methods on our dataset. Through our benchmark, we demonstrate the research gap in cross-modal calibration, the challenges faced by existing methods, and their improved performance after training. This dataset paves the way for the community to develop more accurate, robust, and domain-transferable cross-modal calibration methods.

Primary area:
datasets and benchmarks
Keywords:
Calibration, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_47QxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CalibBench, a diverse camera–LiDAR cross-modal calibration dataset designed to address the limited extrinsic variation and fixed sensor configurations in existing datasets, which hinder the evaluation of calibration methods' generalization capabilities. The authors employ a bimanual robotic platform, a motion capture (MoCap) system, and hand–eye calibration to collect data across indoor and outdoor scenes. The dataset is organized into four categories based on discrete versus continuous traces and static versus dynamic extrinsics. By benchmarking six existing calibration methods, the authors demonstrate substantial performance degradation under diverse extrinsic configurations. Although retraining on CalibBench improves performance, considerable calibration errors remain, highlighting the limitations of existing methods in generalizing across heterogeneous sensor configurations.

Critical strengths
  1. The paper addresses the limited extrinsic diversity of existing camera–LiDAR calibration datasets. Using a dual-arm robotic platform, motion capture (MoCap), and hand–eye calibration, the authors construct a dataset with diverse real-world extrinsic configurations, covering multiple scenes and both static and dynamic calibration settings.
  2. The paper evaluates six existing calibration methods, comparing their original checkpoints with models retrained on CalibBench. The results reveal the limitations of existing methods under diverse sensor configurations and provide useful experimental insights for future research.
Critical weaknesses
  1. The dataset introduces large variations in translation (x, y, z) and rotation (roll, pitch, yaw), which can significantly change scene observations, camera–LiDAR field-of-view overlap, and cross-modal correspondences. However, the experiments do not sufficiently disentangle these factors from extrinsic diversity, making it difficult to determine the primary cause of the observed performance degradation.
  2. CalibBench provides a useful dataset and benchmark. However, scene diversity remains limited, especially since dynamic-extrinsic data are collected only in a single indoor MoCap environment, leaving cross-scene generalization insufficiently validated.
Questions for authors

Q1. Camera–LiDAR Field-of-View Overlap. CalibBench introduces large variations in translation (x, y, z) and rotation (roll, pitch, yaw), which may affect the camera–LiDAR field-of-view overlap and cross-modal correspondences. What is the average field-of-view overlap in CalibBench, and how does its distribution compare with KITTI? Could the authors further analyze the relationship between field-of-view overlap and calibration errors?

Q2. Baseline Reproduction Discrepancies. In Table 6, some reproduced baseline results on KITTI differ substantially from those reported in the original papers. For example, LCCRAFT's translation error increases from 2.9 cm to 11.0 cm. What accounts for these discrepancies? Were the model configurations, training process, and evaluation protocols consistent with the original implementations?

Q3. Ground-Truth Extrinsic Accuracy. The paper primarily validates the estimated extrinsics through reprojection errors, without directly reporting their accuracy in translation and rotation. Could the authors quantitatively compare the extrinsics obtained through MoCap and hand–eye calibration with an independent target-based calibration reference, reporting the corresponding translation and rotation errors to further validate the reliability of the ground-truth extrinsics? Additionally, given that target-based calibration with fiducial markers is often straightforward and highly accurate in indoor robotic settings, it would be useful to include a target-based method as a reference or upper-bound baseline, especially for the indoor experiments.

Q4. Practical Relevance to Robotic Applications. The paper highlights dynamic extrinsic calibration for movable sensors as an important application. However, in many robotic systems, cameras and LiDAR sensors are rigidly mounted on robot arms, heads, or bodies, and their changing poses can often be computed through forward kinematics when the sensor mounting transforms and robot kinematic parameters are known. Could the authors clarify which robotic scenarios specifically require observation-based dynamic extrinsic calibration and what advantages it offers over kinematics-based computation?

Reviewer_YMF3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work presents a camera-lidar extrinsic calibration dataset. The two sensors are mounted on two separate robot arms, so that the sensor extrinsics can be controlled. It enables recording data with dynamic sensor extrinsics controlled by the robot arms, which is the main advantage compared to most existing datasets where the sensors positions are fixed. A motion capture system is used to measure the groundtruth extrinsics. The paper also trains and evaluates some learning-based calibration methods on the proposed dataset.

Critical strengths

The main strength of this work is the recording of continuously varying, controlled extrinsics, which enables evaluation of online calibration.

Critical weaknesses

The work is conceptually valuable but not yet ready for publication. My major concerns are:

  • the groundtruth accuracy claimed in line 150 is not justified. I see the pixel reprojection error in line 199 and the mocap system accuracy in line 672. But these error measurements are not converted to the sensor extrinsic errors.
  • Missing comparisons with existing calibration datasets: PLK-Calib[1] (discrete dynamic extrinsics), STT4AT[2] (continuous dynamic but in simulation), LCScenes from DST-Calib [3] (multiple extrinsic configurations).
  • The discussion of perturbation-based methods and other alternatives is missing. For autonomous driving datasets, like KITTI, Waymo, with fixed sensor setups, the common way acquiring dynamic extrinsics is introducing perturbations to the point cloud [4]. However, the perturbed point clouds are not equivalent to physically moving the sensor (parallax, occlusion, ...). Pairing camera and lidar frames from different timestamps via ego-motion is another way to obtain real viewpoint changes. The benefits of using a real moving sensor pair should be analyzed in details, which would strengthen the motivation of this work.
  • The dataset includes outdoor recordings and will be publicly released under CC BY 4.0. Please state whether identifiable persons and license plates have been anonymized prior to release.

[1]Zhang, Yanyu, et al. PLK-Calib: Single-shot and Target-less LiDAR-Camera Extrinsic Calibration using Plücker Lines. IROS 2025. [2]Zhu, Morui, et al. Mind the Hitch: Dynamic Calibration and Articulated Perception for Autonomous Trucks. CVPR 2026. [3]Z. Huang, et al. , DST-Calib: A Dual-Path, Self-Supervised, Target-Free LiDAR-Camera Extrinsic Calibration Network. IEEE Transactions on Robotics. [4]Iyer, Ganesh, et al. CalibNet: Geometrically supervised extrinsic calibration using 3D spatial transformer networks. IROS 2018.

Questions for authors
  • What is the end-to-end uncertainty of the final LiDAR–camera extrinsics, in rotation and translation?
  • Is GT accuracy validated under motion, or only on static samples? How are lidar motion distortion and rolling shutter (if applicable) handled?
  • Is GT accuracy validated also outdoors? MoCap systems are typically restricted to indoor volumes and degraded by sunlight.
Reviewer_q5AcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CalibBench is a real world lidar-camera calibration dataset, with variation of extrinsics unlike most currently available datasets (ex: driving datasets like KITTI or nuScenes). The camera and lidar are mounted on two robot arms which are moving for each capture. The extrinsic is calculated through motion capture and hand-eye calibration. It has two indoor and four outdoor scenes, and retrains 5 learned calibrators on this data before evaluating them. The results show that the performance of these learned methods is way worse than on datasets with fixed extrinsics, showing that diverse extrinsic calibration remains an open problem.

Critical strengths
  1. Learned calibrators are usually evaluated on a fixed extrinsic rig. This dataset would be as far as I know the first dataset with diverse extrinsic calibration between lidar and camera (BEVCalib also has one called CalibDB but is not released yet). The motion capture plus hand-eye pipeline produce reliable and precise ground truth.
Critical weaknesses
  1. DS frames are from CS sequences, the data is split by sequence and outdoor sequences reuse extrinsics set indoor. So the same extrinsics might appear in both training and test. If the leak exists, the reported errors don't measure generalization.
Questions for authors
  1. Could you check if there is any risk of leakage of extrinsic between train and test due to outdoor extrinsic being set from indoor calibration?
  2. Could you add a row that returns the initial guess as a baseline for calibration?
  3. Koide3 is not a learned method, so it would not memorize the extrinsic and should not perform worse due to diverse extrinsic. Any reason why it fails? Especially on KITTI, where your reported scores (51.4cm/4.5deg) are much worse than BEVCalib (21.1cm/0.6deg) on the same noise level?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.