A Benchmark for Knowing Unknown Things in Multi-Modal Sensor Streams

Reliable autonomous driving requires perception systems to recognize and consistently reason about situations that fall outside their training distribution. Such out-of-distribution (OoD) events are rare by nature, yet can be safety-critical and particularly challenging when objects are partially occluded, dynamically interacting with the scene, or observed differently across sensing modalities. Evaluation therefore requires benchmarks that extend beyond independent, single-modality frames. We introduce Knowing Unknown Things (KnUT), a multi-modal, sequence-based benchmark for evaluating OoD perception and tracking in dynamic driving scenes recorded with a cargo-bike sensor setup. KnUT provides synchronized camera and LiDAR recordings with OoD instances that are consistently identifiable across modalities and over time. The benchmark covers three complementary tasks: pixel-wise OoD detection in camera images, point-wise OoD detection in LiDAR point clouds, and temporal tracking of OoD instances. The data are recorded in live traffic such that OoD objects are routinely occluded by other road users and observed in the far distance. We release an annotated validation set and the complete evaluation code, while keeping the test annotations private for benchmark evaluation. In addition, we benchmark existing OoD methods to provide reproducible reference results. Overall, KnUT provides a unified benchmark for evaluating OoD perception across different perception modalities in both, static frames and aligned sequences.