- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces CertOD, an application of conformal prediction to object detection. The motivation is clear: autonomous driving requires precise classification to define tolerable error margins, which are more critical on some classes than others. Validation is done via coverage evaluation on object detection task on the BDD100K and CODA2022 datasets.
Critical strengths
It is clear that the method works as a wrapper for frozen object detectors that provide multi-class scores. The main idea is explained clearly: use conformal prediction to strive for the predicted class to be correct and the predicted box to contain the labeled box, in an effort to maximize object detection performance, especially for vulnerable road users, such as pedestrians.
Critical weaknesses
The main critical weaknesses concern (1) explanation of background and method and (2) interpretation and relevance of results. The two are explained in detail below with concrete examples of what can be improved:
(1) The paper assumes that the reader is an expert in conformal prediction, as it does not attempt to even briefly explain what it is about, nor any of its central concepts, such as calibration data. The authors could improve readability by introducing conformal prediction and related concepts in the introduction. The work "Ries et al. 2026" that the authors cited provides a good example for how to introduce the topic nicely, and even the remainder of that paper is well written. In the same line, the authors did not cite the work that claims to be the first work to apply conformal prediction to the object detection task: De Grancey, Florence, et al. "Object detection with probabilistic guarantees: A conformal prediction approach." International Conference on Computer Safety, Reliability, and Security. Cham: Springer International Publishing, 2022. Phrases like ”Conformal prediction supplies finite-sample coverage under exchangeability” are unclear. The introduction does not clearly explain terminology, e.g. ”The uncertainty creates a calibration-inference mismatch” is hard to understand. The concept of ”margin” appears quite central to this work, but is not defined nor is any reference provided for it. The introduction reads like a combination of methodology and results sections, as a method overview is given, and some results are already presented; it would be better if the introduction set the background and described the landscape where this work lies, as well as the motivation for this work (which is already somewhat mentioned). The contributions at the end of the introduction read like results, instead of highlighting the method together with its results as a contribution compared to prior art. Some acronyms are not defined, such as CP and IoU. The same goes for some variables: in Section 3.1; M, K, and J are not defined, neither are \Delta^K, x in S(x), \alpha_c, nor box coordinates x and y. Clearly defining these would help the reader understand instead of having them figure them out. The pairing mechanism is not explained; is it Hungarian matching, greedy matching, or something else? It might not be needed to explain it in detail, but at least a reference or a few words would help. From the introduction and Figure 1 I got the feeling that argmax and set-max methods were prior art, but in section 3.2 it seems that they are the authors’ contributions, this should be made clearer. In Section 3.3 it seems that the pair certificate is the main contribution, however, details that are central to understanding the contribution, such as how the enlargement q is done or how many pairs there are, are missing. The certifiability mask and budget are not clearly explained, even though they are central to the contribution.
(2) The experiments do not show comparisons with prior art like Ries et al., 2026 or De Grancey, Florence, et al., 2022, nor is it clearly discussed anywhere how this work contributes additional knowledge to the one that the mentioned prior art contributed. Some phrases appear out of context, such as: ”On BDD100K, the frozen detector misclassifies 1.27% of matched pedestrians.” - which frozen detector are the authors talking about, and where does the number come from? Also, it is not clear if ”The geometric certificate enlarges the detector box outward to contain the true box.” means that the enlargement is a part of the method, or an intended consequence of the method. The citations in line 366 for model and dataset have the wrong order and can confuse readers. Section 4.1 presents results but does not interpret them. Is it positive that Single box shows larger coverage than CertOD? Is the fact that Single box and CertOD have a much larger joint coverage than Nominal a positive result of the method? It is unclear what role the Area metric has in evaluating the method; in particular, is the fact that that the area is larger than in "Nominal" a negative result? Also, I am confused by the bold letters; in Table 3 it would seem that best results are in bold, but the same does not apply in Table 1, this should be clarified. The meaning of "Bad strata" in Table 3 is not defined. In general, the results are just shown but not interpreted, it would be preferred if something was said about the main results and why they are valuable and worth publishing. I didn’t understand why the focus is on the pedestrian class in the conclusion (but also in the abstract and introduction), and other classes are left out of the main thesis; if the focus of the paper was to improve performance for pedestrian object detection, this should be made clearer from the beginning.
Questions for authors
Following the "Critical Weaknesses" section, I would ask the authors to make following main clarifications in the paper: (a) What is conformal prediction, how has it been used in object detection before, and how does your work contribute additional knowledge in that area compared to prior art? (b) How do your results compare to prior art results, and based on the outcome, what is the conclusion? I would recommend the authors to read the "Critical Weaknesses" section for details.