acceptodds
Withdrawn from ICLR 2027

A Benchmark for Knowing Unknown Things in Multi-Modal Sensor Streams

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable autonomous driving requires perception systems to recognize and consistently reason about situations that fall outside their training distribution. Such out-of-distribution (OoD) events are rare by nature, yet can be safety-critical and particularly challenging when objects are partially occluded, dynamically interacting with the scene, or observed differently across sensing modalities. Evaluation therefore requires benchmarks that extend beyond independent, single-modality frames. We introduce Knowing Unknown Things (KnUT), a multi-modal, sequence-based benchmark for evaluating OoD perception and tracking in dynamic driving scenes recorded with a cargo-bike sensor setup. KnUT provides synchronized camera and LiDAR recordings with OoD instances that are consistently identifiable across modalities and over time. The benchmark covers three complementary tasks: pixel-wise OoD detection in camera images, point-wise OoD detection in LiDAR point clouds, and temporal tracking of OoD instances. The data are recorded in live traffic such that OoD objects are routinely occluded by other road users and observed in the far distance. We release an annotated validation set and the complete evaluation code, while keeping the test annotations private for benchmark evaluation. In addition, we benchmark existing OoD methods to provide reproducible reference results. Overall, KnUT provides a unified benchmark for evaluating OoD perception across different perception modalities in both, static frames and aligned sequences.

Primary area:
datasets and benchmarks
Keywords:
Data set, Out-of-distribution detection, Out-of-distribution tracking, Multi-modal, Real world

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iZ7eOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Introducing a dataset and benchmark (KnUT) for OOD detection and early recognition of “unknown objects”. The data is new and collected by the authors using a cargo bike on city streets and parking lots. The sensor setup includes RGB-D, lidar, and radar. The benchmark is both single-frame and multi-frame (tracking based) with ground-truth generated by combining foundation models and human correction. The paper benchmarks 4 camera-based methods and 3 lidar-based methods on KnUT.

Critical strengths

Detecting things that are not necessarily part of the training distribution is indeed an important aspect of the AD perception stack and I think it is important that the field focuses on this aspect of perception stacks.

I like the early recognition metric. The time it takes to react is an important aspect of ADAS, e.g. AEB. That the authors lift this problem setup is good and an important research direction.

Critical weaknesses

Usefulness of the benchmark: It is unclear to me what research gap this benchmark aims to fill. If SAM3 could accurately segment the staged “unknown objects” in these scenes already (it is unclear how much manual effort was added on top of SAM3?), what additional contribution does this benchmark make in terms of opening new research avenues? The paper argues for neural networks and learning-based methods, but doesn’t provide a training set to learn these types of methods, but rather only a validation and test set. What are the benefits of this newly recorded data, on a non-autonomous-vehicle sensor rig, compared to finding interesting cases in and curating existing large-scale AD datasets? Since the sensor setup differs materially from an autonomous vehicle it would be difficult to disentangle what performance degradation/OOD detection from that of change in sensor distribution when training on an AD dataset and using this dataset for evaluating OOD.

Paper story and clarity: The story is a bit hard to follow. The title alludes to a general benchmark for “multi-modal sensor streams” whereas the abstract starts with “Reliable autonomous driving”, and the collection vehicle is a cargo-bike. Together with incoherent text, and figures placement and order of referring to them being shuffled, it makes the paper’s story unclear and the paper feels unpolished. Furthermore, the benchmark’s performance measures are very loosely defined and not clearly motivated, in my opinion two very important aspects of making a benchmark. It would furthermore be interesting to see how learning-based open-vocabulary methods like YOLOE performs on this benchmark in terms of detecting the objects and understanding what gap remains to be closed.

Questions for authors

Why is KnUT useful, what can we measure now that we couldn’t measure before and how do you envision the benchmark being used? What are the motivations behind the selected performance measures?

Reviewer_5C1XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Knowing Unknown Things (KnUT), a multi-modal, sequence-based benchmark for evaluating out-of-distribution (OoD) perception and temporal tracking in dynamic urban environments. To address the limitations of existing OoD datasets that rely on static, single-modality, or synthetic images, KnUT collects synchronized camera and LiDAR sequences aboard a cargo-bike platform in live urban traffic, capturing dynamic occlusions, varying lighting conditions, and remote sensing up to 200 meters. The dataset comprises 14 public validation sequences (1,799 frames) and 71 private test sequences (7,337 frames) densely annotated via a semi-automated pipeline that combines VLTSeg pseudo-labels, SAM3/Labelme manual refinements, and cross-modal ID mapping. KnUT establishes three complementary tasks—2D pixel-wise OoD detection, 3D point-wise OoD detection, and OoD instance tracking—alongside safety-oriented early-recognition metrics including First View and Highest Distance. Benchmarking state-of-the-art methods reveals severe performance drops in temporal tracking and distant early detection, highlighting critical failure modes in current OoD perception models.

Critical strengths
  • KnUT establishes a synchronized multi-modal sequence-based benchmark, bridging a crucial gap by moving beyond single-frame or single-modality evaluation to address temporal tracking, cross-modal alignment, and dynamic occlusions in live traffic. 
  • The paper introduces First View and Highest Distance metrics to evaluate not just whether an anomaly is detected, but how early and from what distance a safe response is possible. 
  • In Section 3.5 and Appendix A.2, the authors develop a practical annotation pipeline integrating VLTSeg pseudo-labeling, SAM3 refinement, boundary artifact removal, and automated face and license plate anonymization.
  • Empirical evaluations demonstrate that top-performing static detection methods collapse under temporal tracking and distant early detection, providing valuable failure modes and research directions for the community.
Critical weaknesses
  • KnUT contains a total of only 9,136 frames across 85 sequences (1,799 val / 7,337 test). This scale is orders of magnitude smaller than standard sequence-based driving benchmarks like the Waymo Open Dataset or nuScenes (hundreds of thousands of frames).
  • The OoD targets in KnUT are created by manually staging 27 specific physical props (e.g., toy cars, lawn ornaments, boxes) across 18 locations. While this ensures controlled physical presence, these staged props feature rigid, relatively simple geometries and static semantic profiles. Unlike naturally occurring real-world corner cases—such as spilled cargo, fallen trees, or irregular construction machinery—the artificially placed props in KnUT lack geometric irregularity, dynamic shape deformation, and natural distribution diversity.
  • Although the paper claims to cover varying weather and lighting conditions, the dataset's visible samples are predominantly limited to clear daylight, sharp tree shadow contrasts, and minor residual snow patches on road edges.
Questions for authors

Given that KnUT is explicitly introduced as a multi-modal benchmark, have the authors evaluated whether joint multi-modal fusion architectures can mitigate the extremely low LiDAR point-level precision and poor distant early-recognition scores?

1 comment
AuthorsOpenReview ↗

Statement on the review

Dear reviewer, thank you for your time and the consideration of our paper. We acknowledge the highlighting of "bridging a crucial gap" and that we introduce a practical annotation pipeline. We would like to clarify some points and answer the questions.

First of all, we would like to clarify, that KnUT is an Out-of-distribution (OoD) dataset. Therefore, it would be unfair to compare it with classical AD datasets like Waymo or nuScence, since the focus of the dataset it different. While Waymo or nuScenes are usable for semantic segmentation in context of AD, KnUT focus on detecting unknown classes. Furthermore, KnUT can be also used for fields like delivery robots, which can also drive on pedestrian areas or bicycle lanes. Compatible datasets are introduced in the related work section, especially Table 1 for a deep comparison of the data sets. Furthermore, we are the first complex OoD dataset with multi-modal approach, containing a tracking branch. We acknowledge that using a predefined set of staged objects limits the diversity of object geometries and does not cover the full range of naturally occurring road hazards. However, the objects are physically placed in real urban environments and recorded under varying conditions, including different positions, distances, orientations, occlusions, and both static and moving scenarios. Thus, while the object categories are predefined, their spatial and temporal contexts reflect real-world conditions. Staging objects is a deliberate trade-off that allows us to create reproducible scenarios and ensure sufficient representation of OoD objects, while avoiding the safety risks and practical difficulties of recording naturally occurring hazards. We will extend the evaluation with an ablation study across more detailed scenario categories, considering both the OoD objects and their surroundings. These categories will include indoor/outdoor objects, remote-sensing scenarios, occlusion, and standard OoD scenarios. This analysis will provide further insights into how different object and scene characteristics affect detection performance. In addition, we will extend Figure 10 in the appendix with examples from the test split to better illustrate the diversity of KnUT.

Multi-modal fusion is one usable approach to solve KnUT. We will introduce a zero-shot fusion method for point-clouds and images in the new version of the paper. However, this will be done for the detection only. We will let the researchers open to introduce such complex models, to solve the tasks. Therefore, we provide this multi-modal data.

Reviewer_uj3jOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This is a dataset/benchmark paper for the driving arena. The emphasis is on knowing unknown things and data is collected via camera, depth camera, LiDAR and RADAR. Benchmarks offered are pixel-wise OoD detection for camera, point-wise OoD detection in LiDAR clouds, and baselines for OoD temporal tracking. Two new metrics, First v[ew and Highest distance are contributed.

Critical strengths

Researchers can now begin to assess the early recognition properties of their algorithms.

Critical weaknesses

The early recognition work could have gone deeper; for example, could have benchmarked whether some classes of OoD behaviors/objects, are easier to recognize early.

Questions for authors

What additional contribution can this paper include towards benchmarking early recognition? If the paper can include at least one more metric or exploration of the early recognition area, the paper would then make a substantial contribution to Learning Representations.

1 comment
AuthorsOpenReview ↗

Statement on the review

Dear reviewer, thank you for your time and the consideration of our paper. We acknowledge the highlighting of the early-recognition. However, this paper does not only focus on a new metric but the main point is to provide a new much more complex, multi-modal Out-of-distribution (OoD) dataset which is closer to the reality of AD. Furthermore, with recording not with a vehicle but with a bicycle, it opens the world to use this OoD dataset not only for autonomous driving of cars but also in context of delivery robots for example. We will make this point clearer in the paper.

Regarding early recognition, existing OoD benchmarks primarily rely on pixel- and segment-level OoD detection metrics per frame. Temporal datasets additionally employ established tracking metrics to evaluate object tracking over time. To the best of our knowledge, however, no existing benchmark specifically evaluates how early OoD objects can be recognized in a temporal setting. Our benchmark addresses this gap by introducing dedicated early-recognition metrics that assess the temporal aspect of OoD detection. We agree that further analysis of this aspect would strengthen the paper. To evaluate more detailed, how methods behave for different OoD objects, we will introduce more categories of the sequences e.g. weather, lightning condition, ground, moving objects. This serve as a more detailed evaluation as well as a proof, that our data set is much more complex in those categories. To show more the diversity, we will also extend Figure 10 of the Appendix with test data. Furthermore, for a better overview, we will change Table 2 and Table 3 and add a column calls "Early-Recognition" instead of combine it with OoD tracking.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.