- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Introducing a dataset and benchmark (KnUT) for OOD detection and early recognition of “unknown objects”. The data is new and collected by the authors using a cargo bike on city streets and parking lots. The sensor setup includes RGB-D, lidar, and radar. The benchmark is both single-frame and multi-frame (tracking based) with ground-truth generated by combining foundation models and human correction. The paper benchmarks 4 camera-based methods and 3 lidar-based methods on KnUT.
Critical strengths
Detecting things that are not necessarily part of the training distribution is indeed an important aspect of the AD perception stack and I think it is important that the field focuses on this aspect of perception stacks.
I like the early recognition metric. The time it takes to react is an important aspect of ADAS, e.g. AEB. That the authors lift this problem setup is good and an important research direction.
Critical weaknesses
Usefulness of the benchmark: It is unclear to me what research gap this benchmark aims to fill. If SAM3 could accurately segment the staged “unknown objects” in these scenes already (it is unclear how much manual effort was added on top of SAM3?), what additional contribution does this benchmark make in terms of opening new research avenues? The paper argues for neural networks and learning-based methods, but doesn’t provide a training set to learn these types of methods, but rather only a validation and test set. What are the benefits of this newly recorded data, on a non-autonomous-vehicle sensor rig, compared to finding interesting cases in and curating existing large-scale AD datasets? Since the sensor setup differs materially from an autonomous vehicle it would be difficult to disentangle what performance degradation/OOD detection from that of change in sensor distribution when training on an AD dataset and using this dataset for evaluating OOD.
Paper story and clarity: The story is a bit hard to follow. The title alludes to a general benchmark for “multi-modal sensor streams” whereas the abstract starts with “Reliable autonomous driving”, and the collection vehicle is a cargo-bike. Together with incoherent text, and figures placement and order of referring to them being shuffled, it makes the paper’s story unclear and the paper feels unpolished. Furthermore, the benchmark’s performance measures are very loosely defined and not clearly motivated, in my opinion two very important aspects of making a benchmark. It would furthermore be interesting to see how learning-based open-vocabulary methods like YOLOE performs on this benchmark in terms of detecting the objects and understanding what gap remains to be closed.
Questions for authors
Why is KnUT useful, what can we measure now that we couldn’t measure before and how do you envision the benchmark being used? What are the motivations behind the selected performance measures?
Statement on the review
Dear reviewer, thank you for your time and the consideration of our paper. We acknowledge the highlighting of "bridging a crucial gap" and that we introduce a practical annotation pipeline. We would like to clarify some points and answer the questions.
First of all, we would like to clarify, that KnUT is an Out-of-distribution (OoD) dataset. Therefore, it would be unfair to compare it with classical AD datasets like Waymo or nuScence, since the focus of the dataset it different. While Waymo or nuScenes are usable for semantic segmentation in context of AD, KnUT focus on detecting unknown classes. Furthermore, KnUT can be also used for fields like delivery robots, which can also drive on pedestrian areas or bicycle lanes. Compatible datasets are introduced in the related work section, especially Table 1 for a deep comparison of the data sets. Furthermore, we are the first complex OoD dataset with multi-modal approach, containing a tracking branch. We acknowledge that using a predefined set of staged objects limits the diversity of object geometries and does not cover the full range of naturally occurring road hazards. However, the objects are physically placed in real urban environments and recorded under varying conditions, including different positions, distances, orientations, occlusions, and both static and moving scenarios. Thus, while the object categories are predefined, their spatial and temporal contexts reflect real-world conditions. Staging objects is a deliberate trade-off that allows us to create reproducible scenarios and ensure sufficient representation of OoD objects, while avoiding the safety risks and practical difficulties of recording naturally occurring hazards. We will extend the evaluation with an ablation study across more detailed scenario categories, considering both the OoD objects and their surroundings. These categories will include indoor/outdoor objects, remote-sensing scenarios, occlusion, and standard OoD scenarios. This analysis will provide further insights into how different object and scene characteristics affect detection performance. In addition, we will extend Figure 10 in the appendix with examples from the test split to better illustrate the diversity of KnUT.
Multi-modal fusion is one usable approach to solve KnUT. We will introduce a zero-shot fusion method for point-clouds and images in the new version of the paper. However, this will be done for the detection only. We will let the researchers open to introduce such complex models, to solve the tasks. Therefore, we provide this multi-modal data.