- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents CalibBench, a diverse camera–LiDAR cross-modal calibration dataset designed to address the limited extrinsic variation and fixed sensor configurations in existing datasets, which hinder the evaluation of calibration methods' generalization capabilities. The authors employ a bimanual robotic platform, a motion capture (MoCap) system, and hand–eye calibration to collect data across indoor and outdoor scenes. The dataset is organized into four categories based on discrete versus continuous traces and static versus dynamic extrinsics. By benchmarking six existing calibration methods, the authors demonstrate substantial performance degradation under diverse extrinsic configurations. Although retraining on CalibBench improves performance, considerable calibration errors remain, highlighting the limitations of existing methods in generalizing across heterogeneous sensor configurations.
Critical strengths
- The paper addresses the limited extrinsic diversity of existing camera–LiDAR calibration datasets. Using a dual-arm robotic platform, motion capture (MoCap), and hand–eye calibration, the authors construct a dataset with diverse real-world extrinsic configurations, covering multiple scenes and both static and dynamic calibration settings.
- The paper evaluates six existing calibration methods, comparing their original checkpoints with models retrained on CalibBench. The results reveal the limitations of existing methods under diverse sensor configurations and provide useful experimental insights for future research.
Critical weaknesses
- The dataset introduces large variations in translation (x, y, z) and rotation (roll, pitch, yaw), which can significantly change scene observations, camera–LiDAR field-of-view overlap, and cross-modal correspondences. However, the experiments do not sufficiently disentangle these factors from extrinsic diversity, making it difficult to determine the primary cause of the observed performance degradation.
- CalibBench provides a useful dataset and benchmark. However, scene diversity remains limited, especially since dynamic-extrinsic data are collected only in a single indoor MoCap environment, leaving cross-scene generalization insufficiently validated.
Questions for authors
Q1. Camera–LiDAR Field-of-View Overlap. CalibBench introduces large variations in translation (x, y, z) and rotation (roll, pitch, yaw), which may affect the camera–LiDAR field-of-view overlap and cross-modal correspondences. What is the average field-of-view overlap in CalibBench, and how does its distribution compare with KITTI? Could the authors further analyze the relationship between field-of-view overlap and calibration errors?
Q2. Baseline Reproduction Discrepancies. In Table 6, some reproduced baseline results on KITTI differ substantially from those reported in the original papers. For example, LCCRAFT's translation error increases from 2.9 cm to 11.0 cm. What accounts for these discrepancies? Were the model configurations, training process, and evaluation protocols consistent with the original implementations?
Q3. Ground-Truth Extrinsic Accuracy. The paper primarily validates the estimated extrinsics through reprojection errors, without directly reporting their accuracy in translation and rotation. Could the authors quantitatively compare the extrinsics obtained through MoCap and hand–eye calibration with an independent target-based calibration reference, reporting the corresponding translation and rotation errors to further validate the reliability of the ground-truth extrinsics? Additionally, given that target-based calibration with fiducial markers is often straightforward and highly accurate in indoor robotic settings, it would be useful to include a target-based method as a reference or upper-bound baseline, especially for the indoor experiments.
Q4. Practical Relevance to Robotic Applications. The paper highlights dynamic extrinsic calibration for movable sensors as an important application. However, in many robotic systems, cameras and LiDAR sensors are rigidly mounted on robot arms, heads, or bodies, and their changing poses can often be computed through forward kinematics when the sensor mounting transforms and robot kinematic parameters are known. Could the authors clarify which robotic scenarios specifically require observation-based dynamic extrinsic calibration and what advantages it offers over kinematics-based computation?