acceptodds
Under review as a conference paper at ICLR 2027

Which OOD Detector Should You Deploy When Test-Set Gains Disagree?

Abstract

A deployed out-of-distribution (OOD) detector meets unfamiliar inputs whose mix is unknown and changing. Its effectiveness is highly sensitive to this mix. Yet on benchmarks, detectors that gain on some test sets perform poorly on others, sometimes below a vanilla cross-entropy baseline. How should one choose a detector whose performance holds across the mixes it may meet? We find that these losses are reproducible effects of training. Predictions fixed before training hold across seeds, scoring functions, architectures and datasets, with and without auxiliary OOD data. We compare a new detector with every existing choice, including the baseline under each scoring function, and compute in closed form how far the mix can shift before it stops being the top performer. At the published means of four benchmarks, the baseline wins under this comparison for about half of the training gains measured under a shared scoring function. None of those gains holds under every mix. A detector that passed this comparison held an 8.8-point lead in area under the ROC curve on new images and halved false acceptances. Gains that failed it fell short on new runs and images. OOD detectors should therefore be chosen by their stability over plausible mixes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
out-of-distribution detection, outlier exposure, OpenOOD, benchmark evaluation, post-hoc scorers, model selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iyQ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper first introduces a finding that the performance of OOD detectors varies across different datasets. Thus, the paper aims to identify reliable OOD detectors. The paper introduces a metric to evaluate the AUROC gain and proposes that a new detector should replace the current detector when the new detector can beat the old detector on both mixed distributions with given source weights and with different source weights. Also, the new OOD detector should meet the required AUROC gain and ID accuracy. Different OOD detectors are evaluated with the proposed metric and protocol. The experiments show that none of the tested detectors' gains hold under every mix

Critical strengths
  1. The paper proposes to identify reliable OOD detectors, which I think is meaningful in this area

  2. The metric seems mathematically reasonable.

Critical weaknesses
  1. The whole paper is like a combination of fragments. It is really hard to follow the authors' logic. For instance, in line 174, a passage suddenly appears explaining networks.

  2. Even though the metric is sound in mathematics, it can not test the detector in a real OOD distribution, since it requires that the test distribution be available to identify weights ww. Even though OOD detectors cannot access these seen test distributions, the whole system remains within the seen loop.

Questions for authors

OE methods are trained on OOD data, so they can be more reliable than OOD detection methods. But in Table 5, CE+RDMS or CE+GEN can lead. Is there any explanation for this phenomenon?

5 comments
AuthorsOpenReview ↗

Response to Reviewer iyQ7 (1 of 3)

We thank you for your comments and question. We are encouraged that you find identifying reliable OOD detectors meaningful in this area, and the metric mathematically reasonable. Section and table numbers refer to the revised PDF.

W1. The whole paper is like a combination of fragments. It is really hard to follow the authors' logic. For instance, in line 174, a passage suddenly appears explaining networks.

The revision ties the paper's three parts to one comparison: when the mix of test sets can shift over a specified range, a new detector is valuable only if it beats every existing choice across that range.

  • (a) Why it is needed. A training gain on one OOD test set often comes with a loss on another, and that loss recurs under retraining (Figure 1, Section 4.1). The better detector then depends on the mix of test sets (Section 2).
  • (b) What it changes in published results. About half of 39 published near-OOD training gains disappear against the cross-entropy baseline's best scorer (Section 4.3).
  • (c) Which gains pass it on new data. With CIFAR-10 as in-distribution data, Outlier Exposure (OE) keeps a 9.2-point lead in area under the ROC curve (AUROC) on new images in a replication fixed before scoring (Section 4.1).

The introduction now states (a) to (c), a roadmap closes Section 1, and Section 3 opens with an overview of the experiments in Section 4's order. The passage you flagged at line 174 now sits in Section 3.1, as the definition the OE loss uses, and we audited the entire paper for similar loose fragments.

AuthorsOpenReview ↗

Response to Reviewer iyQ7 (2 of 3)

W2. Even though the metric is sound in mathematics, it can not test the detector in a real OOD distribution, since it requires that the test distribution be available to identify weights ww. Even though OOD detectors cannot access these seen test distributions, the whole system remains within the seen loop.

We understand "seen loop" to mean the measured sources, and "real OOD distribution" to mean the inputs a detector meets at deployment.

The revised title says "compare" where it said "deploy": the comparison is defined on the measured sources. It needs each OOD source to be identified and measured, and a reference mix stated, but no estimate of the weights ww, because it evaluates every mix of those sources at once (Section 2).

No evaluation can measure every source a deployment may meet, so data can only probe your concern, using images that played no part in finding a lead, under protocols fixed before scoring. The revision probes one source outside the measured sources, iNaturalist, where the leading detector keeps its lead by 6.47 AUROC points (Section 3.2, Appendix H.4; the table's first row). We also ran two tests with fresh data, but within the measured sources (Section 4.1, Appendices H.2 and H.3; rows 2 and 3). The leads held in each. The detector is Outlier Exposure with maximum softmax probability (OE with MSP), where CIFAR-10 or CIFAR-100 is the in-distribution (ID) data. Its rivals are 14 vanilla choices: the cross-entropy baseline under seven scorers and two settings regimes.

New test dataWhat it shows
A source outside the measured set (iNaturalist, with CIFAR-10 as ID, Appendix H.4)The lead carried to one unmeasured source. OE with MSP leads there by 6.47 AUROC points (lower bound 6.21)
New ID and OOD images from the same sources (CIFAR-10.1 as ID, Appendix H.2)The lead is not an artifact of the benchmark images. It is at least 9.17 AUROC points at equal weights (one-sided 95% lower bound 8.57), and a breakdown computed afterwards finds OE ahead on both sources, so no mix reverses the lead
Fresh OOD images (CIFAR-100 as ID, Appendix H.3)The boundary computed on the benchmark predicts where the lead ends on new images. OE leads until CIFAR-10 makes up 93.4 percent of the near-OOD inputs on the benchmark, and 93.8 percent on the new images
AuthorsOpenReview ↗

Response to Reviewer iyQ7 (3 of 3)

Q. OE methods are trained on OOD data, so they can be more reliable than OOD detection methods. But in Table 5, CE+RDMS or CE+GEN can lead. Is there any explanation for this phenomenon?

In Table 5 (now Table 7), CE+RMDS and CE+GEN lead only on two sheets where OE was not compared. Where all three are compared, OE leads in most cases.

Table 5. We follow the presentation of OpenOOD v1.5. Its two ImageNet-1K sheets, for ViT-B-16 and Swin-T, report only post-hoc scorers on a model trained with cross-entropy (CE) alone, so OE is absent there. Following your question, a note in Table 7 now says so.

The comparison you asked about. We added it to Appendix B and repeat the table below. It covers the four OpenOOD sheets that report all three detectors. OE with maximum softmax probability (MSP) leads in seven of eight comparisons on the near-OOD test sets, and in five of eight over all test sets. A rival leads in four comparisons, all on ImageNet-200.

The explanation you asked for. Training on auxiliary OOD images helps on some test sets and costs on others, so OE's lead depends on which test sets count. Section 2's comparison locates where each lead ends.

Sheet (OE's AUROC lead over the rival in points, test sets weighted equally; negative means the rival leads)Near-OOD test sets: vs CE+RMDSvs CE+GENAll test sets: vs CE+RMDSvs CE+GEN
CIFAR-10+5.0+6.6+4.2+5.3
CIFAR-100+8.1+7.0+1.7+3.5
ImageNet-200+2.3+1.2+1.5−0.9
ImageNet-200, full-spectrum (shifted images of familiar classes added to the ID data)−2.0+5.0−4.3−0.8
Reviewer_o6r2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how OOD detector rankings change with the mixture of OOD sources. It shows that OE’s gains and losses across sources can recur across seeds, scorers, architectures, and datasets. The authors use AUROC’s linearity in source weights to calculate when a detector’s lead over existing choices disappears, then apply this analysis to published benchmarks and selected new-image evaluations.

Critical strengths
  1. The paper addresses a practical limitation of reporting results on a single OOD test set.

  2. The mixture analysis is clear, mathematically grounded, and easy to reproduce from per-source AUROCs.

  3. Experiments cover multiple seeds, scorers, datasets, and architectures; negative results are reported transparently.

  4. The paper carefully distinguishes AUROC mixture analysis from source-specific FPR@95.

Critical weaknesses
  1. The main analytical contribution builds on a straightforward AUROC identity and established optimization tools; the methodological novelty is limited.

  2. The “new images” evaluation uses separate samples from already measured sources, so it does not establish generalization to unseen OOD sources.

  3. The proposed robustness measure depends on the chosen source set, reference mixture, and allowed shifts. These choices remain difficult to determine for deployment.

  4. Most uncertainty analyses are conditional on fixed evaluation sets, and published-table sensitivity checks rely on assumptions about noise and dependence.

Questions for authors

Can the authors test sensitivity to adding or omitting OOD sources?

Can they clarify that the new-image results concern new samples from measured sources, not unseen OOD source generalization?

3 comments
AuthorsOpenReview ↗

Response to Reviewer o6r2 (1 of 3)

We thank you for a close reading and specific questions. We are glad you find the problem practical, the mixture analysis clear, grounded, careful and reproducible, and the experiments broad and transparent. Section and table numbers refer to the revised PDF.

W1. The main analytical contribution builds on a straightforward AUROC identity and established optimization tools; the methodological novelty is limited.

We contribute a principled methodology with implications for reporting and reading OOD evaluations over the full range of mixes. We build it with standard tools, which keep it exact and reproducible from per-source AUROCs, as you note. The issue is pressing. Detector rankings are inconsistent across datasets, seeds and splits (Tajwar et al., 2021; Szyc et al., 2023). The methodology has three implications:

  1. Gains can be reported with the mixes where they hold. Any published per-source table gives them in closed form, with no new experiments (Section 2).
  2. Conclusions from published results change. About half of 39 published near-OOD training gains disappear against the cross-entropy baseline's best scorer, and none holds under every mix (Section 4.3).
  3. What passed this stricter test held up. A CIFAR-10 gain over the baseline under every scorer, on every measured source, held on new images and on one source outside the measured set, in tests fixed before scoring (W2 and W4 below).

W2. The "new images" evaluation uses separate samples from already measured sources, so it does not establish generalization to unseen OOD sources.

Q2. Can they clarify that the new-image results concern new samples from measured sources, not unseen OOD source generalization?

Correct: the new images come from the measured sources, and the paper states its claims for mixes of those sources. Section 2 now says that the third condition, the gain test on separate images, uses new images from the measured sources, and the abstract reads "on new images from its two near-OOD sources". These images are new samples of the same sources: category-matched images, screened against every earlier image (Appendices H.2 and H.3). For CIFAR-10 the in-distribution (ID) images are new too (CIFAR-10.2 and CIFAR-10.1).

The revision also adds one test on a source outside the measured set, and the CIFAR-10 lead of Outlier Exposure with maximum softmax probability (OE with MSP) holds there by 6.47 AUROC points (95% lower bound 6.21). iNaturalist, OpenOOD's set of plant photographs, lies outside CIFAR-10's six measured test sets, and none of its images shows a CIFAR-10 class or comes from ImageNet, OE's auxiliary source. The test was fixed before scoring (Appendix H.4). The lead is over the best of 14 vanilla choices: the cross-entropy baseline under seven scorers, with settings selected on TinyImageNet or on Textures.

AuthorsOpenReview ↗

Response to Reviewer o6r2 (2 of 3)

W3. The proposed robustness measure depends on the chosen source set, reference mixture, and allowed shifts. These choices remain difficult to determine for deployment.

This concern follows from your point on generalization (W2 and Q2). The revised title says "compare" where it said "deploy": the three choices are made for the measured sources. No evaluation can measure every source a deployment may meet, so the paper does not set them for a deployment. Beyond the measured sources, data can only probe, as the iNaturalist test in W2 does. Within them, the analysis in Q1 below asks the same question in miniature: it leaves a measured source out of the evaluation, or adds one, and reports whether the leader changes. Table 3 does the same for the reference mixture and allowed shift.

Q1. Can the authors test sensitivity to adding or omitting OOD sources?

Yes. We ran both changes, add and omit, on all eight published AUROC sheets, and the radius says in advance which of them cannot change the leader: all 17 changes inside a leader's radius kept it. We thank you for suggesting this analysis, new in the paper (Section 4.3, Table 4, Appendix B).

Why these changes. Omitting a source asks what an evaluation made before that source was measured would have concluded. Adding one asks what a source measured later, as at deployment, would change.

What we learned. The radius gives a rule of thumb: a leader is safe against losing any one of KK sources when its radius exceeds 1/K1/K, and against any one added far source when it exceeds 1/31/3. The radius is the smallest change in source weights at which a rival ties the leader. On six of the eight sheets the leaders fall short of both marks. Of the 76 changes, 57 keep the leader, and all 19 that replace it fall outside a radius.

The table gives four of the eight sheets: the leader's radius, the size of the change, and how many changes kept the leader. We committed the protocol before computing any change.

SheetOmit one source: leader's radiusSize of the changeLeader keptAdd far sources to the near pair: leader's radius (over all sources, so it can exceed 0.5)Leader kept
CIFAR-100.1380.1676 of 60.0063 of 5
CIFAR-1000.0230.1673 of 60.3204 of 5
ImageNet-2000.0970.2003 of 50.1601 of 4
ImageNet-1K0.4420.2005 of 5, guaranteed0.8404 of 4, guaranteed
AuthorsOpenReview ↗

Response to Reviewer o6r2 (3 of 3)

W4. Most uncertainty analyses are conditional on fixed evaluation sets, and published-table sensitivity checks rely on assumptions about noise and dependence.

We tested both: fixed evaluation sets, on new images, and assumed noise and dependence, with measured correlations.

Fixed evaluation sets. The CIFAR-10 lead and the CIFAR-100 boundaries reproduce on new images. The CIFAR-10.1 replication (Appendix H.2) replaces ID and OOD images, and OE with MSP leads all 14 vanilla choices by at least 9.17 AUROC points at equal weights (95% lower bound 8.57, with images resampled). The CIFAR-100 tests (Appendix H.3) replace the OOD images and reproduce every tested boundary within 0.011 in source weight.

Noise and dependence. With measured correlations in place of the assumed ones, each leader checked stays first about as often, within two percentage points. Our matched ten-seed runs keep every seed, so they look beyond the means and measure how errors correlate (Appendix G). The published sheets give at most one standard deviation per cell. The checks assumed independent errors, or one error per training method and source. On CIFAR-100 and CIFAR-10, the correlations are 0.82 and 0.89 among one model's scorers, between the assumed 0 and 1, and 0.16 and 0.20 among one detector's sources, above the assumed 0. The table repeats the checks with the measured values (CIFAR values for the ImageNet-200 sheets).

Leader testedStays first under the two assumptionsStays first with measured correlations
CIFAR-100, leader over all sources, at the published spread (the largest move on the CIFAR-10, CIFAR-100 and standard ImageNet-200 sheets)88 to 89% of resampled tables87%
ImageNet-200 full-spectrum, near-pair leader, at the published spread98%96%; with the CIFAR-100 values, the 5th percentile of its tie radius falls from about 0.08 to 0.035, still above zero
CIFAR-100, near-pair leader, at our measured seed spread (1.73 times the published)100%100%
CIFAR-100, leader over all sources, at that spread74%72%
Reviewer_f9fnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how to choose an out-of-distribution (OOD) detector when gains on one test set come with losses on another. A detector is a trained classifier plus a scorer. For a fixed in-distribution (ID) population, AUROC against a mixture of OOD sources equals the corresponding weighted average of per-source AUROCs (Lemma 1, Appendix A). From that identity the authors compute, in closed form, the total-variation shift in source weights at which a leader ties or falls behind a rival (Eq. 3, Definition 1).

Definition 1 says a new detector should replace the current ones only if it beats every existing choice, including the baseline under every scorer, at a stated reference mixture; keeps that lead under a shift written down in advance; and meets a gain and ID-accuracy requirement on separate images. Empirically, matched retraining shows that Outlier Exposure (OE) losses on a named source recur across ten CIFAR-100 seeds, seven scorers, WideResNet-40-2, and five ImageNet-200 seeds, and that LogitNorm shows a similar CIFAR-10 loss without auxiliary OOD data (Section 4.1). On four OpenOOD v1.5 tables, 19 of 39 near-OOD training gains remain ahead of cross-entropy under its best scorer, and none leads every such scorer on every source (Section 4.3). A blend frozen before a separate-image evaluation fails the prespecified 0.5-point rule (Section 4.4). An OE configuration that is ahead on every measured CIFAR-10 source is reported to keep an 8.81-point AUROC lead on separate images (Appendix H.1).

Critical strengths
  1. The source losses are shown as training effects rather than as one unlucky split. On CIFAR-100 with ResNet-18, OE minus cross-entropy is negative on CIFAR-10 for all seven scorers across ten new seeds (Figure 1), the signs survive Holm correction, six of seven losses survive on WideResNet-40-2, and the ImageNet-200 iNaturalist loss was predicted before five seeds and then reproduced on five further runs (−3.31 MSP points against a 0.44 threshold; Section 4.1). LogitNorm, with no auxiliary OOD data, lowers CIFAR-10 AUROC by 4.2 points and ID accuracy by 2.4. That is consequential because the rest of the paper is about whether such conflicts should change a deployment choice, and the conflicts are not artifacts of a single run.

  2. The decision object matches the metric. Lemma 1 makes the leader, the tie weight, and the total-variation radius functions of the per-source AUROC table, and the authors apply that table to 178 published candidates: only 26 are mixture-admissible, and 19 of 39 near-average training gains still beat cross-entropy under its best published scorer (Section 4.3). Comparing a new method with the baseline’s best scorer, rather than with the baseline under the same scorer, changes the ImageNet-200 near-OOD lead from 2.06 over vanilla MSP to 0.44 over vanilla Scale. This is the comparison a practitioner who can still use the old detector actually faces.

Critical weaknesses
  1. The result used to show that a gain which passes Definition 1 holds on new images was not a blinded confirmation. Appendix H.1 says the CIFAR-10 OE-versus-vanilla read-out used saved outputs, and that separate-image point estimates at weight 0.65 were already visible, which is why it is labeled a read-out. The protocol that was committed before scoring separate images tested the blend and did not meet the requirement fixed for that evaluation: the AUROC margin on CIFAR-10.2 is 0.46 points, with image-bootstrap range [0.244, 0.645], below the prespecified 0.5-point lower bound, and the ID-accuracy cost is 0.647 points (Section 4.4). The abstract nevertheless leads with an 8.8-point lead and halved false acceptances. Contribution 3, that the definition separates gains that hold from gains that fall short on new data, is carried by the failures and by a success whose estimates had already been seen. That is enough to keep the paper out of clear acceptance until the abstract and Section 4.1 state the same status as Appendix H.1.

  2. The deployment claim is a reweighting of the benchmark sources already in the table. Total variation is computed between weight vectors, and the paper notes that this need not equal the variation distance between mixtures when sources overlap (Appendix A). No experiment estimates a deployment mixture, and the selection exercises “limit the use of measured-source comparisons as selection policies for new sources” (Section 4.3). The title and the last sentence of the conclusion still instruct practitioners to choose detectors by stability over the mixes they may meet. The evidence supports reporting a radius on the measured sources. It does not support a selector for an unseen mix. Where no allowed shift was stated before the radius was computed, Definition 1’s second condition is explicitly left open (Section 2), so several “meets the second condition for any allowed shift below the radius” statements do not apply a constraint the authors committed to in advance.

Questions for authors
  1. In Appendix H.1, which of the three Definition 1 quantities for CIFAR-10 OE with MSP were computable from the candidate table before that read-out protocol was committed, and which, if any, were not yet visible?

  2. Is there a registered comparison for which a numerical allowed shift was written down before r0r_0 was computed and for which the new detector’s radius exceeded that shift? If every reported second condition is of the form “the radius is rr, so any shift below rr would pass,” what would falsify condition 2?

2 comments
AuthorsOpenReview ↗

Response to Reviewer f9fn (1 of 2)

We thank you for your detailed review of our work, especially for checking the abstract against Appendix H.1. We are glad you find the reproducible losses consequential, and the comparison with the baseline's best scorer the one practitioners face. Section and table numbers refer to the revised PDF.

W1. The result used to show that a gain which passes Definition 1 holds on new images was not a blinded confirmation. Appendix H.1 says the CIFAR-10 OE-versus-vanilla read-out used saved outputs, and that separate-image point estimates at weight 0.65 were already visible [...] The abstract nevertheless leads with an 8.8-point lead and halved false acceptances. [...] That is enough to keep the paper out of clear acceptance until the abstract and Section 4.1 state the same status as Appendix H.1.

Thank you for stating what would resolve this. Section 4.1 now states, as Appendix H.1 does, that the read-out's separate-image estimates had been read before its protocol was fixed. The abstract reports a replication fixed before scoring in its place.

The read-out. Section 4.1 calls the 8.8-point result "a read-out of saved outputs", adding: "Every candidate's point estimates on those images, at the blend's frozen weight, had been printed and read when the read-out's protocol was fixed." Contribution 3 drops "separates gains that hold from gains that fall short".

The replication repeats the comparison, with the same checkpoints and settings, on CIFAR-10.1 as ID and fresh CIFAR-100 and TinyImageNet images.

The lead replicates: the smallest gain is 9.17 points in the replication and 8.81 in the read-out, and false acceptances are halved in both. The replication establishes the gain requirement, its primary outcome, and the abstract says the lead "met the gain requirement of a replication fixed before scoring". Its accuracy cost is reported without a verdict: 1,600 ID images cannot settle a 0.5-point ceiling unless the cost is near zero. The read-out meets that ceiling.

The table compares the two, against the 14 vanilla choices at equal near-OOD weights.

OutcomeRead-out, CIFAR-10.2 (H.1)Replication, CIFAR-10.1 (H.2)
Protocol fixedafter every candidate's estimates at weight 0.65 had been readbefore any image was scored
Smallest AUROC gain (one-sided 95% lower bound)8.81 (8.52)9.17 (8.57)
ID-accuracy cost (upper bound)0.07 (0.23)0.14 (0.54)
False acceptance, OE with MSP vs the vanilla choice with the lowest rate34.8% vs 70.4%34.6% vs 72.9%
ID images accepted, same pair94.7% vs 94.3%96.3% vs 95.6%

Q1. In Appendix H.1, which of the three Definition 1 quantities for CIFAR-10 OE with MSP were computable from the candidate table before that read-out protocol was committed, and which, if any, were not yet visible?

From the candidate table alone, only third-condition point estimates at weight 0.65 were computable: each candidate's separate-image AUROC there, and the accuracy cost. The equal-pair gain behind the verdict, and every bound, were first computed under the protocol. Outside that table, some inputs to the first two conditions had been printed. Your question prompted an addition to the paper: Appendix H.1 now states, for each condition, what had been printed and what was first computed. The read-out's protocol "was committed before any margin or bound of this comparison was computed".

The table separates, for each quantity, what printed values already implied from what was first computed under the protocol.

Definition 1 quantityPrinted before the protocol, and what it impliesNot yet visible: first computed under it
1. Margin (original images)Both models' near-source AUROCs at default settings (Table 13), which imply the smallest near-source margins. Far-OOD means (Appendix D.3)Margins on each far source, and against the four scorers whose settings Textures reselects
2. Boundary (original images)Implied by row 1: no near-pair boundary against the seven default-settings choicesNo boundary against any of the 14 choices
3. Gain and accuracy requirement (separate images)The candidate table: each candidate's AUROC at weight 0.65 and ID accuracy, hence the accuracy cost, with ID acceptance and false acceptance at that weightEqual-pair AUROC and false-acceptance contrasts, and every bound
AuthorsOpenReview ↗

Response to Reviewer f9fn (2 of 2)

W2. The deployment claim is a reweighting of the benchmark sources already in the table. [...] The evidence supports reporting a radius on the measured sources. It does not support a selector for an unseen mix.

The revised title says "compare" where it said "deploy": the comparison is defined on the measured sources. As you write, the evidence supports reporting a radius on them. The conclusion's last sentence, which you also named, now reads "by their stability over plausible mixes of the measured sources", as does the abstract.

The paper states the limits you cite, and they stand. The radius is a distance between source-weight vectors, which bounds the distance between the mixtures and need not equal it when sources overlap (Appendix A). The practitioner states the reference and the allowed shift, and no experiment estimates a deployment mixture (Section 2). Our selection rules show no general benefit on withheld sources, which limits measured-source comparisons as selection policies for new sources (Section 4.3).

One test, fixed before scoring, goes outside the measured set. On iNaturalist, CIFAR-10 OE with MSP keeps its lead over the best of 14 vanilla choices by 6.47 AUROC points (95% lower bound 6.21, Appendix H.4). It is one source, and the claims stay on the measured ones.

The second-condition statements you flagged are removed: the paper claims condition 2 only at a shift stated in advance (Q2 below).

Q2. Is there a registered comparison for which a numerical allowed shift was written down before r0r_0 was computed and for which the new detector's radius exceeded that shift? If every reported second condition is of the form "the radius is rr, so any shift below rr would pass," what would falsify condition 2?

Yes. On a third network, ResNet-34, a shift of 0.25 was registered before any ResNet-34 run existed, and the radius, 0.397, exceeded it. The shift admits every near-pair mix from 25/75 to 75/25. The radius is that of CIFAR-100 OE with MSP (TinyImageNet-597 pool) against the 14 vanilla choices, with 90% interval 0.387 to 0.411 (Appendix H.5). The shift was chosen knowing the ResNet-18 radius and WideResNet-40-2 boundaries, so the test is prospective for this network only.

Condition 2 fails when the radius is at or below a shift stated before that radius is computed. The paper claims condition 2 only at such a shift and otherwise reports the boundary (Section 2).

The same shift fails a benchmark leader. On ResNet-18, OE with the 300K pool leads at the reference, but on fresh images scored after the shift was fixed its radius is 0.225. By the point-estimate rule fixed in advance, it fails condition 2 (Appendix H.3). Its 90% interval, 0.157 to 0.294, contains 0.25, and its benchmark radius, 0.215, was known.

A candidate ahead on every measured source, such as CIFAR-10 OE with MSP, has no boundary. New images could fail it only by creating one inside the shift.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision (revised PDF uploaded)

We thank the reviewers for their time, comments and suggestions. We are encouraged by the reviewers' view that the problem is meaningful, practical and consequential for practitioners' choices (iyQ7, o6r2, f9fn). Reviewers found the methodology mathematically reasonable and grounded, careful and easy to reproduce, with a decision object matched to the metric (iyQ7, o6r2, f9fn). They found the analysis clear and the negative results reported transparently (o6r2). They also commended our experiments across multiple seeds, scorers, datasets and architectures, which show the source losses as training effects (o6r2, f9fn).

The importance of our work lies in the methodology it proposes for reporting, comparing and choosing OOD detectors based on how far the mix of the measured OOD sources can shift. Its central insight is that, because existing detectors remain available, what matters about a gain is the full range of mixes over which the new detector beats every existing choice, including the baseline under each scorer. Below we summarize the key contributions:

  • Gains can be reported with the mixes where they hold. From a per-test-set table, the comparison gives those mixes in closed form (o6r2, f9fn).
  • Conclusions from published results change. Against the baseline under its best scorer, about half of the published near-OOD training gains on four benchmarks disappear, and none holds under every mix (f9fn).
  • What passed this stricter test held up. A detector that leads on every measured source kept a 9.2-point AUROC lead on new images from its two near-OOD sources in a replication fixed before scoring, and that lead met its gain requirement. Two other gains fell short of that requirement on new runs and images. Detectors can therefore be chosen by their stability over plausible mixes of the measured sources.

We changed the title for one reason: "deploy" suggested a claim about the mix a detector meets in use, and the paper's comparison is over mixes of the measured sources (f9fn W2). The title now asks the question the paper's methodology answers, "How Should You Compare OOD Detectors When Test-Set Gains Disagree?" (submitted as "Which OOD Detector Should You Deploy When Test-Set Gains Disagree?"). No result or analysis changes with it. The abstract, the introduction and the conclusion now say explicitly what Section 2 defines, a comparison over the measured sources (f9fn W2; o6r2 W2, Q2). The abstract's headline result is now a replication fixed before scoring, a 9.2-point lead on new images. Section 4.1 now describes the earlier 8.8-point result as Appendix H.1 does, a read-out of saved outputs (f9fn W1). Presentation changes are listed in iyQ7's thread. The revision adds seven analyses:

QuestionResponds toStatusNew analysis and resultWhere
Does a detector that leads on every measured source keep its lead on images no candidate had scored?f9fn W1; o6r2 W4; iyQ7 W2ReplicatedCIFAR-10.1 replication fixed before scoring. Outlier Exposure with maximum softmax probability (OE with MSP) leads the baseline under every scorer by at least 9.17 AUROC points at equal near-OOD weights§4.1, App. H.2
Does such a lead hold on one source outside the measured ones?iyQ7 W2; o6r2 W2, Q2; f9fn W2HeldiNaturalist, protocol fixed before scoring. The same detector leads by at least 6.47 pointsApp. H.4
Do the decision boundaries reappear on new OOD images?f9fn Q2; o6r2 W4; iyQ7 W2ReproducedCIFAR-100 on fresh OOD images. Every tested boundary, the mix at which a lead ends, reappears within 0.011 in source weight§4.1, App. H.3
Does a lead hold over a shift fixed before the models were trained?f9fn Q2ConfirmedResNet-34 on CIFAR-100, with an allowed shift of 0.25 registered before any ResNet-34 run. OE with MSP's lead ends at a shift of 0.397, beyond the registered shift§4.1, App. H.5
Is the leader guaranteed when a source is omitted or added?o6r2 Q1, W3Guaranteed within the radiusAll eight published AUROC sheets. All 17 changes smaller than the leader's radius, the shift at which a rival ties it, keep the leader. Of the 59 larger changes, 19 replace it§4.3, Table 4, App. B
Do the published-table checks hold under measured correlations?o6r2 W4VerifiedMatched ten-seed runs measure the correlations the checks assumed. With them, the leaders stay first nearly as often as under the assumed ones (for instance, on CIFAR-100, 87 percent of tables against 88 to 89)§4.3, App. G
Why do CE+RMDS and CE+GEN lead in Table 7 (the submission's Table 5)?iyQ7 QExplainedThey lead only on the two sheets that report post-hoc scorers alone (ViT-B-16, Swin-T). On the four sheets with all three detectors, OE with MSP leads in seven of eight near-OOD comparisons and five of eight over all test setsTable 7, App. B

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.