- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper first introduces a finding that the performance of OOD detectors varies across different datasets. Thus, the paper aims to identify reliable OOD detectors. The paper introduces a metric to evaluate the AUROC gain and proposes that a new detector should replace the current detector when the new detector can beat the old detector on both mixed distributions with given source weights and with different source weights. Also, the new OOD detector should meet the required AUROC gain and ID accuracy. Different OOD detectors are evaluated with the proposed metric and protocol. The experiments show that none of the tested detectors' gains hold under every mix
Critical strengths
-
The paper proposes to identify reliable OOD detectors, which I think is meaningful in this area
-
The metric seems mathematically reasonable.
Critical weaknesses
-
The whole paper is like a combination of fragments. It is really hard to follow the authors' logic. For instance, in line 174, a passage suddenly appears explaining networks.
-
Even though the metric is sound in mathematics, it can not test the detector in a real OOD distribution, since it requires that the test distribution be available to identify weights . Even though OOD detectors cannot access these seen test distributions, the whole system remains within the seen loop.
Questions for authors
OE methods are trained on OOD data, so they can be more reliable than OOD detection methods. But in Table 5, CE+RDMS or CE+GEN can lead. Is there any explanation for this phenomenon?
5 comments
Response to Reviewer iyQ7 (2 of 3)
W2. Even though the metric is sound in mathematics, it can not test the detector in a real OOD distribution, since it requires that the test distribution be available to identify weights . Even though OOD detectors cannot access these seen test distributions, the whole system remains within the seen loop.
We understand "seen loop" to mean the measured sources, and "real OOD distribution" to mean the inputs a detector meets at deployment.
The revised title says "compare" where it said "deploy": the comparison is defined on the measured sources. It needs each OOD source to be identified and measured, and a reference mix stated, but no estimate of the weights , because it evaluates every mix of those sources at once (Section 2).
No evaluation can measure every source a deployment may meet, so data can only probe your concern, using images that played no part in finding a lead, under protocols fixed before scoring. The revision probes one source outside the measured sources, iNaturalist, where the leading detector keeps its lead by 6.47 AUROC points (Section 3.2, Appendix H.4; the table's first row). We also ran two tests with fresh data, but within the measured sources (Section 4.1, Appendices H.2 and H.3; rows 2 and 3). The leads held in each. The detector is Outlier Exposure with maximum softmax probability (OE with MSP), where CIFAR-10 or CIFAR-100 is the in-distribution (ID) data. Its rivals are 14 vanilla choices: the cross-entropy baseline under seven scorers and two settings regimes.
| New test data | What it shows |
|---|---|
| A source outside the measured set (iNaturalist, with CIFAR-10 as ID, Appendix H.4) | The lead carried to one unmeasured source. OE with MSP leads there by 6.47 AUROC points (lower bound 6.21) |
| New ID and OOD images from the same sources (CIFAR-10.1 as ID, Appendix H.2) | The lead is not an artifact of the benchmark images. It is at least 9.17 AUROC points at equal weights (one-sided 95% lower bound 8.57), and a breakdown computed afterwards finds OE ahead on both sources, so no mix reverses the lead |
| Fresh OOD images (CIFAR-100 as ID, Appendix H.3) | The boundary computed on the benchmark predicts where the lead ends on new images. OE leads until CIFAR-10 makes up 93.4 percent of the near-OOD inputs on the benchmark, and 93.8 percent on the new images |
Response to Reviewer iyQ7 (3 of 3)
Q. OE methods are trained on OOD data, so they can be more reliable than OOD detection methods. But in Table 5, CE+RDMS or CE+GEN can lead. Is there any explanation for this phenomenon?
In Table 5 (now Table 7), CE+RMDS and CE+GEN lead only on two sheets where OE was not compared. Where all three are compared, OE leads in most cases.
Table 5. We follow the presentation of OpenOOD v1.5. Its two ImageNet-1K sheets, for ViT-B-16 and Swin-T, report only post-hoc scorers on a model trained with cross-entropy (CE) alone, so OE is absent there. Following your question, a note in Table 7 now says so.
The comparison you asked about. We added it to Appendix B and repeat the table below. It covers the four OpenOOD sheets that report all three detectors. OE with maximum softmax probability (MSP) leads in seven of eight comparisons on the near-OOD test sets, and in five of eight over all test sets. A rival leads in four comparisons, all on ImageNet-200.
The explanation you asked for. Training on auxiliary OOD images helps on some test sets and costs on others, so OE's lead depends on which test sets count. Section 2's comparison locates where each lead ends.
| Sheet (OE's AUROC lead over the rival in points, test sets weighted equally; negative means the rival leads) | Near-OOD test sets: vs CE+RMDS | vs CE+GEN | All test sets: vs CE+RMDS | vs CE+GEN |
|---|---|---|---|---|
| CIFAR-10 | +5.0 | +6.6 | +4.2 | +5.3 |
| CIFAR-100 | +8.1 | +7.0 | +1.7 | +3.5 |
| ImageNet-200 | +2.3 | +1.2 | +1.5 | −0.9 |
| ImageNet-200, full-spectrum (shifted images of familiar classes added to the ID data) | −2.0 | +5.0 | −4.3 | −0.8 |
Response to Reviewer iyQ7 (1 of 3)
We thank you for your comments and question. We are encouraged that you find identifying reliable OOD detectors meaningful in this area, and the metric mathematically reasonable. Section and table numbers refer to the revised PDF.
The revision ties the paper's three parts to one comparison: when the mix of test sets can shift over a specified range, a new detector is valuable only if it beats every existing choice across that range.
The introduction now states (a) to (c), a roadmap closes Section 1, and Section 3 opens with an overview of the experiments in Section 4's order. The passage you flagged at line 174 now sits in Section 3.1, as the definition the OE loss uses, and we audited the entire paper for similar loose fragments.