acceptodds
Under review as a conference paper at ICLR 2027

Tuning OOD Detectors Without Given OOD Data

Abstract

Many existing out-of-distribution (OOD) detectors are tuned using a separate dataset deemed OOD with respect to the training distribution, often ad hocly chosen. In practical settings, such OOD data may be unavailable or difficult to obtain; even when available, they may not represent actual OOD data, which often include "unknown unknowns.” As such, there is significant variance in detector performance depending on the ad hoc OOD dataset used for tuning. Hence, we introduce and formalize the often neglected problem of tuning OOD detectors without a "given” OOD dataset. We propose and evaluate two strong baseline methods based on related literature and found that they too yield inconsistent performance across detectors and datasets. We address this by introducing simulated hold-out tuning (SHOT), a generic approach that requires no additional data other than the ID training set. SHOT repeatedly holds out subsets of ID classes and treats them as simulated OOD samples for detector tuning. We evaluate these approaches on six OOD detectors across the OpenOOD and BROAD datasets, spanning both semantic and covariate shifts. SHOT achieves the highest number of wins across datasets and detectors compared with the baselines, particularly excelling at higher parameter detectors. These results show that effective OOD detector tuning need not rely on collecting a separate OOD dataset, removing a significant practical barrier to OOD detection in real-world deployments.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Out of distribution detection, tuning OOD detectors

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aURuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the problem of hyperparameter tuning for OOD detectors when no separate OOD tuning dataset is available. The authors argue that existing OOD detection benchmarks often rely on arbitrarily selected OOD validation datasets, which may not represent the unknown distributions encountered during deployment. They first investigate two alternative tuning strategies based on Gaussian noise and adversarial perturbations. To address the limitations of these approaches, the paper proposes Simulated Hold-Out Tuning (SHOT), which repeatedly partitions the available ID classes into held-in and held-out subsets. Surrogate classifiers are trained on the held-in classes, while the held-out classes serve as simulated OOD examples. OOD detector hyperparameters are then selected through Bayesian optimization over these simulated detection tasks.

Critical strengths
  1. The proposed SHOT framework is conceptually intuitive and detector-agnostic. Using held-out ID classes to construct simulated OOD detection tasks provides a natural way to exploit existing labeled data without collecting external OOD examples.
  2. SHOT achieves encouraging results, especially for detectors with larger hyperparameter spaces, c.f., Table 3.
  3. This is a meaningful problem because representative OOD data are often unavailable in real deployments.
Critical weaknesses
  1. Firstly, high comp cost. The paper reports that training one ImageNet-1K ResNet-50 surrogate takes approximately 20–24 hours using four RTX 3090 GPUs. With 30 surrogate networks in the full configuration, the aggregate computation is substantial.
  2. The most essential problem of this paper, about its core idea, removing classes from training can substantially change the learned representation, classification head, feature geometry, and confidence distribution. Even when only a small number of classes is removed, there is no general guarantee that the resulting surrogate decision boundary remains close to that of the original network.
  3. The benchmark and evaluation paradigm are quite old-school. Please consider more modern setting beyond the classification task, which is extensively studied few years ago. Moreover, the selected classification tasks in your main exp are also naive.
  4. The performance looks incremental and trivial. In several experiments, SHOT improves AUROC by only a small fraction of a percentage point, potentially within the reported standard deviation.
Questions for authors
  1. Could the authors compare SHOT with computationally inexpensive ID-only tuning methods?
  2. Could the authors report the average and median AUROC improvements, along with statistical significance, instead of relying primarily on the number of wins?
  3. Could the authors provide a more detailed end-to-end cost–performance comparison (would be better with the code evidence)?
3 comments

W1. High computational cost …

Response: The computational cost is incurred entirely offline, with no additional inference overhead, and surrogate networks can be reused across multiple OOD detectors. Table 5 demonstrates that reducing the number of surrogate networks often achieves comparable AUROC on ImageNet-1K. Furthermore, Appendix G estimates the full SHOT computation at approximately 1,300 USD, substantially below the illustrative 35,000 USD – 50,000 USD cost of collecting OOD data. Importantly, unlike collecting known anomalies, SHOT does not assume access to representative future OOD data.

W2. Even when only a small number of classes is removed, there is no general guarantee that the resulting surrogate decision boundary remains close to that of the original network

Response: There may be a gap between the boundary recovered and the true ID/OOD boundary; SHOT can still be used and the error rate is determined by the gap size. Given that the image statistics match between held out and held in, the gap is generally small (but this can depend on the particular dataset as the reviewer notes). Empirical out-performance of our method over competing methods suggests the gap in SHOT is smaller than competing methods. We will clarify the geometric intuition in Section 4 to avoid implying that an exact boundary approximation is required.

W3. The benchmark and evaluation paradigm are quite old-school. Please consider more modern setting beyond the classification task, which is extensively studied few years ago …

Response: Our focus is on the fundamental problem of OOD detector tuning without given OOD data, rather than developing OOD detectors for specific tasks. Classification provides a standardized and rigorous setting through OpenOOD and BROAD, enabling systematic evaluation across six detector families and diverse semantic and covariate shifts, including ImageNet-1K. Extending SHOT to other tasks would require task-specific adaptations and evaluation protocols, constituting a separate research problem beyond the scope of this paper. The significance of the tuning problem is independent of the complexity of the underlying prediction task.

W4. The performance looks incremental and trivial. In several experiments, SHOT improves AUROC by only a small fraction of a percentage point, potentially within the reported standard deviation.

Response: We disagree that SHOT's improvements are merely incremental or trivial. To quantify their magnitude, we analyzed AUROC gains over the better of Gaussian and adversarial tuning across 180 evaluations. SHOT achieves improvements of at least 1 AUROC point in 36 cases, at least 2 points in 25 cases, and at least 5 points in 5 cases, with a maximum gain of 8.56 points. Thus, improvements are substantial in many settings, even if modest in others. More importantly, SHOT's primary contribution is enabling effective OOD detector tuning without any given OOD data, rather than merely maximizing AUROC gains. These results demonstrate that SHOT can provide meaningful improvements while eliminating reliance on externally supplied OOD tuning data (Table 4).

Q1. Could the authors compare SHOT with computationally inexpensive ID-only tuning methods?

Response: Gaussian and Adv are computationally inexpensive ID only tuning. We additionally compared to inexpensive augmentations of ID data as suggested by Reviewer 6T6c (see Response to W3 of Reviewer 6T6c) - SHOT is better.

Q2. Could the authors report the average and median AUROC improvements, along with statistical significance, instead of relying primarily on the number of wins?

Response: Please see Response to W4.

Q3. Could the authors provide a more detailed end-to-end cost–performance comparison (would be better with the code evidence)?

Response: Please refer to L418-425 of the manuscript (“Computational Cost”). We will release the code after acceptance.

I appreciate the efforts and time from the authors for the additional explaination.

However, considering both novelty and empirical advantage, I'm sorry I can not be more positive, and will keep my current rating.

We would appreciate it if the reviewer could explicitly identify which aspects of our methodological contribution lack novelty, with reference to relevant prior work. Regarding empirical advantage, we ask the reviewer to clarify the scientific basis for deeming our empirical contributions insufficient, particularly in light of the extensive evidence across 180 detector-OOD configurations presented in our global response/final remarks, demonstrating a breadth of generalization across diverse OOD distributions that none of the competing tuning approaches consistently achieves.

Reviewer_6T6cOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper explores for the first time of using ID samples only to optimize for the posthoc scoring hyperparameters. This is a first study of its kind to the best of my knowledge. SHOT trains a number of proxy models. SHOT hides a few classes and trains on the rest of the classes. The hidden classes are the OOD for the model. Model is trained in many settings with few classes hidden in each. A candidate hyperparameter is choosen and tested on every proxy models. In this way, the hyperparameter search takes place and the hyperparameters that yields best average performance is then finalized for the final model to evaluate the final OOD detection performance.

Critical strengths

This paper truly respects that "unknown is unknown" theme of OOD detection by just tuning the parameters only on the ID data. Hence, the OOD data is not required to tune the parameters of scoring methods. The performance gain with respect to the gaussian noise and adversarial perturbation is non trivial.

Critical weaknesses

The number of proxy models that need to be trained for determining hyperparameters is high. Hence, this method exhibits high computational overhead. In deployment setting, it can be argued that curating small set of curated dataset could be easier than tuning the large number of proxy models.

One of the concern is scaling issue?

How would the method scale when the number of classes increases?

Is this method feasible for datasets like ImageNet-1k datasets ?

Could the authors provide the results with pixel shuffling operation ? For example, just shuffling 10% of the pixels of the image?

If such in-image augmentation could perform similarly then this elaborate process of training surrogate model is questionable.

Also, please provide the comparison with operations such as jigsaw shuffle / patch shuffle.

The authors have not done thorough experiments with within-image augmentation. Gaussian noise is easily separable and adversarial perturbation can be nearly invisible. The authors are encouraged to show extensive comparisons with baselines which corrupt ID images hence lose the semantics but keep the image statistics of the image.

Examples that need to be compared to:

  • Rotation?
  • Mixup / CutMix operations

and similar more operations.

I believe if the proposed method doesn't substantially outperform these methods, the computational cost brought by the method is questionable. Cause those operations are almost free of cost.

Questions for authors

See weakness.

11 comments

W1 The number of proxy models ... is high … it can be argued that curating small set could be easier than tuning the large number of proxy models.

Response: Please see Appendix G (page 18), where be have compared the cost of SHOT to curated data collection; the analysis favors SHOT. Moreover, as argued in L53-72: despite having a tuning set, whether collected or from a publicly available source, one would remain uncertain regarding the performance of the OOD detector,” as there is no guarantee that collected OOD samples adequately cover the space of possible OOD inputs.

W2. How would the method scale when the number of classes increases? Is this method feasible for datasets like ImageNet-1k datasets

Response: Yes it is feasible for ImageNet-1k, in fact our benchmarking includes CIFAR-10/100, ImageNet-200, ImageNet-1k. The number of proxy networks does not necessarily need to increase with the number of classes (in fact our results on ImageNet-200 and ImageNet-1k used about the same number of proxy networks). We have also shown cost ablations for ImageNet-1k (Table 5) - performance versus number of proxy models - it shows the performance remains relatively stable between 1 and 30 proxy networks.

W3. The authors have not done thorough experiments with within-image augmentations.

Response: Within-image augmentations do not necessarily generate meaningful OOD detector tuning sets. Whether a transformation produces OOD data depends on the invariances of the model. For example, rotation may preserve the ID representation for a rotation-invariant model, while patch shuffling may remain ID for architectures insensitive to global spatial arrangements. Similarly, MixUp and CutMix do not guarantee that the resulting samples lie outside the learned ID distribution (a cut mix image may be a valid ID example if similar background or correlated object is pasted and blended). Thus, these augmentations are not necessarily effective surrogates for unknown OOD data.

Nevertheless, to address the reviewer's concern, we conducted extensive additional experiments comparing SHOT against rotation, MixUp, CutMix, pixel shuffling, and jigsaw transformations, alongside Gaussian noise and adversarial perturbations (FGSM), across OOD detectors (ASH, PLF, ReAct, ViM, VRA) and 13 OOD datasets on the ImageNet-1k/ResNet-50 benchmark.

Table 1. Average AUROC (%) ± standard deviation across five OOD detectors.

Tuning methodNear-OODFar-OODBROAD-5ImageNet-C/V2/ROverall
Gaussian noise73.14 ± 4.3992.90 ± 1.1465.92 ± 1.2575.90 ± 0.3775.56 ± 1.34
FGSM75.34 ± 2.6791.42 ± 2.8566.15 ± 1.2376.10 ± 0.4875.69 ± 1.25
SHOT77.30 ± 1.3392.96 ± 1.3566.28 ± 1.1876.13 ± 0.1376.41 ± 0.90
Pixel shuffle74.44 ± 3.0693.89 ± 1.6164.42 ± 1.7875.10 ± 1.1075.23 ± 1.23
Jigsaw72.45 ± 4.8392.42 ± 2.0862.94 ± 2.5973.81 ± 2.3673.72 ± 2.45
Rotation74.58 ± 2.9491.91 ± 2.8563.84 ± 2.0475.01 ± 1.0174.55 ± 1.83
MixUp72.33 ± 5.2489.82 ± 4.0661.91 ± 3.4973.54 ± 2.4872.64 ± 3.35
CutMix70.33 ± 7.3684.46 ± 15.8060.98 ± 4.5272.05 ± 5.1570.39 ± 7.45

Standard deviations are calculated across the five OOD detectors. Datasets are equally weighted within each benchmark group, and the overall mean is computed across all 13 individual datasets.

Table 2. Number of wins across detector–dataset combinations (highest AUROC).

Tuning methodASHPLFReActViMVRATotal
Gaussian noise4150212
FGSM102429
SHOT4846527
Pixel shuffle2313312
Jigsaw100001
Rotation211015
MixUp110002
CutMix100001

65 detector–dataset comparisons; tied methods each receive a win.

The results establish three findings:

  1. Best overall performance: SHOT achieves the highest mean AUROC (76.41%) and leads in three of four benchmark groups.

  2. Broad effectiveness: SHOT achieves 27 wins, more than twice any competitor (at most 12), including 8 for PLF, 6 for ViM, and 5 for VRA.

  3. Consistency: SHOT achieves the lowest overall standard deviation (0.90), indicating more consistent aggregate performance across detectors.

These experiments demonstrate that inexpensive within-image augmentations do not match SHOT's overall effectiveness or consistency.

We substantially expanded our experiments to include Pixel Shuffle as an additional OOD tuning baseline (as other augmentations are not competitive). Across 180 evaluations, SHOT achieves 80 wins (44.4%), compared with 45 (25.0%) for Pixel Shuffle, 33 for adversarial perturbations, and 22 for Gaussian noise. SHOT leads on four of the five benchmark groups (tying on ImageNet-1K), including 26/48 wins on ImageNet-1K covariate shifts versus only 6/48 for Pixel Shuffle. These results reinforce SHOT’s effectiveness and generality relative to alternative ID-image perturbation strategies.

BenchmarkGaussianAdv.Pixel ShuffleSHOT
CIFAR-10171117
CIFAR-100541215
ImageNet-20048612
ImageNet-1K461010
ImageNet-1K (Cov.)88626
Total22334580

The improvement is not significant enough as compared to other baselines for me to improve the score.

In the Table 1 the authors provide in the rebuttal: the improvement over the closest baseline is not even 1% in AUROC. Introducing such computational cost for that trivial gain is easy to argue against.

This makes it very difficult for me to raise the score.

Could the authors also report FPR score too?

Reviewer_WAEDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the problem of lacking OOD samples in OOD detection by using parts of the ID data as (fake)-OOD data. The advantage is the removal of dependence on the choice of OOD dataset when calibrating hyper-parameters of OOD algorithms, because OOD dataset is hard to retrieve and the performance also depend on the particular choice of OOD dataset, making the results unreliable. To achieve the goal, the algorithm first split the training classes into ID and OOD tuning sets, and fit the model using the ID tuning set and fit the OOD detection hyperparameter with the OOD tuning set. With multiple trials, one can aggregated them together for the selection of the final hyper-parameter, and apply to the actual model. Experiments on CIFAR and ImageNet using several popular OOD methods this hyperparameter tuning is effective in many settings, compared to the Gaussian OOD samples or the Adversarial OOD samples.

Critical strengths
  1. The method provides a way to tune hyperparameters of OOD methods without using actual OOD samples. This is the major advantage of the method.
Critical weaknesses
  1. The method requires substantial computation because multiple models need to be trained, which makes it not practical for larger models.
  2. The effectiveness of the method seems to be less impressive for larger models and larger datasets. For example, on ImageNet-1K + ResNet 50, the proposed hyper-paramter tuning only surpasses Gaussian and Adversarial in 2 of the 5 OOD methods evaluated (Tab. 1).
  3. Results on ViTs are missing. So it is unclear if the method generalizes to different model architectures.
Questions for authors
  1. The availability of OOD set enables the tuning of more hyperparameters than some methods originally designed. For example, the matching coefficient α\alpha (Eq. 6) in ViM was manually designed. But it is possible to make it a tunable parameter. Could you please try to make this parameter tunable in your framework to see if better results can be reached. This may bring more value to the proposed method.
  2. Fig. 2(c) is misleading. The ID/OOD boundary by SHOT clearly classifies ID samples as OOD. I do not think this is intended. From my understanding, it is the hyperparameter that transfers from dummy tuning sets to the real model, not those decision boundaries.
3 comments

W1. method requires substantial computation … multiple models need to be trained … not practical for larger models

Response: We disagree that computational cost makes SHOT impractical. The cost is incurred entirely offline, with no additional inference overhead, and surrogate networks can be reused across multiple OOD detectors. Furthermore, as an empirical result, Table 5 shows that the results are fairly insensitive to the number of seeds for held out class selection and the choice of the number of classes to leave out (M) for the detectors examined on ImageNet-1k, while Appendix D (Tables 6–7) shows limited sensitivity to holdout sizes. Therefore, if computational cost is a constraint, one can decrease the number of surrogates significantly and achieve comparable results.

More fundamentally, representative OOD data may be unavailable or prohibitively expensive to collect, with no guarantee of covering future unknown distributions. A curated set of known anomalies does not resolve the open-world problem of detecting future unknowns. SHOT replaces this uncertain data-collection process with predictable offline computation, whose cost is quantified and compared with data acquisition in Appendix G; we reduce total cost by 27×–38×.

W2. effectiveness .. seems to be less impressive for larger models/datasets … on ImageNet-1K + ResNet 50, proposed tuning only surpasses Gaussian and Adversarial in 2 of the 5 OOD methods evaluated

Response: We disagree that SHOT's effectiveness diminishes at scale. Table 1 evaluates six, not five, detectors. As summarized in Table 3, on ImageNet-1K, SHOT is best or tied-best on 4 of 6 detectors, including 4/5 wins for PLF, the detector with the most hyperparameters (7). On ImageNet-1K covariate shifts, SHOT achieves 28/48 wins, versus 10/48 each for Gaussian and adversarial tuning.

More importantly, SHOT's objective is competitive performance without given OOD tuning data. Table 4 demonstrates performance comparable to tuning with given OOD data, sometimes outperforming it. These results support SHOT's effectiveness at ImageNet scale while eliminating dependence on external OOD tuning data.

W3. Results on ViTs are missing. unclear if the method generalizes …

Response: Our experiments follow the standardized OpenOOD benchmark, which specifies ResNet architectures rather than ViTs for the evaluated protocols. SHOT is architecture-agnostic by construction, as its surrogate training and hyperparameter tuning procedures do not depend on CNN-specific properties. Thus, the absence of ViT experiments reflects the choice of a standardized evaluation benchmark, not an architectural restriction of SHOT.

Q1. The availability of OOD set enables the tuning of more hyperparameters than some methods originally designed … the matching coefficient in ViM .. make this parameter tunable ..

Response: We conducted additional experiments on ImageNet-1K/ResNet-50 (table below), jointly tuning ViM's matching coefficient α\alpha and subspace dimension using SHOT. The results demonstrate that tuning α\alpha improves AUROC on 8 of 13 OOD datasets, including 6 of 8 covariate-shift datasets. The overall mean AUROC increases from 75.81% to 76.20%. Notable improvements include Textures (+6.18), ImageNet-C (+3.26), and AutoAttack (+1.90 percentage points). These results further demonstrate SHOT's flexibility in optimizing additional detector hyperparameters, including parameters originally selected heuristically, without requiring any given OOD data.

Results: ViM with Joint Tuning of α\alpha and Subspace Dimension (AUROC %, ImageNet-1K/ResNet-50)

MethodSSB-hardNINCOiNaturalistTexturesOpenImage-OImageNet-CImageNet-V2ImageNet-RAutoAttackPGDStable Diff.Cond. BigGANMulti-LabelOverall Mean
ViM (SHOT)71.8881.7894.1191.4090.9782.0658.8787.6984.6960.8475.0338.3267.9475.81
ViM with α\alpha (SHOT)68.4180.0591.0497.5891.5185.3258.0888.9686.5961.7176.6839.0465.6576.20

Overall Mean is the unweighted average AUROC across all 13 OOD datasets. Bold indicates the better result for each dataset.

Q2. Fig. 2c is misleading. ID/OOD boundary by SHOT classifies ID samples as OOD.

Response: The inclusion of some misclassified ID samples in Fig. 2(c) is intentional, illustrating that SHOT approximates rather than perfectly recovers the ideal ID/OOD boundary. The figure illustrates how multiple held-out class selections approximate different portions of this boundary. We agree that detector hyperparameters, not surrogate decision boundaries, are transferred to the original network, as explicitly described in Section 4 and Algorithm 2.

Thanks for the rebuttal. The extra experiment on tuning α\alpha of ViM does look interesting and promising. However, I'm not convined by the response of other questions.

  1. The cost comparison of collecting data vs. training models is not convincing. a) The proposed method relies on data in the training set, so if people collect OOD sets that look similar to the training set instead of seeking for very expensive OOD data, then the cost is much more manageble, and the proposed method is not expected to be much better than it. b) The training cost may grow when using larger model, and it may cost more than outsourceing for some datasets.
  2. The question on the generalization to ViT is unrelated to how a benchmark like OpenOOD is designed.
  3. Fig. 2c is misleading and I still cannot get the idea of "By holding out multiple sets of random classes, SHOT approximates other parts of the ideal ID/OOD boundary". The red boundary looks a bad choice because it explicitely classify ID samples as OOD. My understanding still be that the hyperparameter transfers, but not the boundaries.

1a) Visual similarity to training data does not establish that samples are OOD: a network may generalize to such samples and effectively treat them as ID. More importantly, Figure 1 shows that tuning with arbitrarily collected OOD datasets produces widely varying detector performance. Thus, such data offers no assurance of effective tuning; success depends on which dataset happens to be chosen.

1b) This is true, as discussed in the limitations section. Our method, like any other, has limitations, and if one cannot afford the cost, they would seek an alternative. There are clearly many instances when our method is more advantageous in cost or when the cost can be tolerated for performance. We offer that option to the user.

2) We agree that generalization to ViTs and the design of OpenOOD are separate issues. Our use of OpenOOD provides standardized, reproducible comparisons across established OOD detectors and architectures. SHOT's formulation is not tied to CNNs or any particular backbone. While evaluating ViTs would further establish architectural generalizability, its absence does not diminish the extensive empirical evidence supporting SHOT under the evaluated settings.

3) The red boundary does not classify ID samples as OOD; the reviewer may have overlooked that "ID held out" is not ID but OOD to the network trained on the held-in data.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Comment to All Reviewers

We urge the reviewers to look beyond the offline tuning cost and consider the alternative to SHOT, which is to actively collect a dataset for tuning OOD detectors. As discussed in L48-70, L413-425, Sec. 6 and 7, and Appendix F and G, it has been established that data collection is expensive and collecting OOD data (often rare events) is even more so. Our analysis in Appendix G places data collection costs to be between 34,000 USD to 50,000 USD; SHOT reduces such costs to a fixed amount of 1,300 USD, which is 27 to 38 times less. In practice, even if one were given an OOD tuning dataset, it is difficult to ascertain whether it sufficiently covers the OOD distribution (Figure 1). SHOT addresses this broader problem by approximating the ID OOD boundary using ID data, which is logically the key ingredient to detect OOD data, as OOD data must be defined with respect to an ID distribution. We will make both of these points clear in the next revision.

AuthorsOpenReview ↗

Final Remarks to the Area Chair and Reviewers

We thank the reviewers for their feedback. We believe the two main concerns are (1) computational cost relative to perceived marginal gains, and (2) the lack of guarantees on generalization to unknown OOD distributions. We emphasize that our contributions go beyond performance improvements (which we have justified are substantive) and address a fundamental challenge in OOD detection.

1. A Fundamental and Practically Important Problem

To our knowledge, this work provides the first comprehensive empirical demonstration of effective OOD detector tuning without any given OOD data, a critical problem for real-world deployment, where the nature of future OOD samples is inherently unknown.

We evaluate 180 detector–OOD configurations across four ID benchmarks, six detectors, multiple architectures, and diverse OOD datasets.

2. A Principled Contribution, Not Another Heuristic

SHOT introduces a fundamental design principle: because unknown OOD samples may arise anywhere outside the ID distribution, tuning should aim to approximate the ID/OOD boundary. By using multiple randomly held-out class subsets and surrogate networks, SHOT aims to approximate different regions of the ID/OOD boundary.

3. Compelling New Evidence of Generalization, Including Reviewer-Suggested Baselines

The table provides striking evidence of SHOT's generalizability across fundamentally diverse OOD distributions: semantic shifts, natural distribution shifts, corruptions, adversarial attacks, and synthetic data. The reviewer-suggested Pixel Shuffle and our Gaussian noise baseline each outperform SHOT on only 1/18 OOD dataset families; even our strongest baseline, adversarial tuning, does so on only 4/18. Moreover, adversarial tuning outperforms Pixel Shuffle on 14/18 families, confirming the strength of our original baselines.

These results support the design principle illustrated in Figure 2(c): SHOT aims to approximate different regions of the ID/OOD boundary, enabling generalization to unforeseen OOD distributions rather than specializing to particular shifts. This breadth of generalization is the fundamental contribution of SHOT.

\def\G#1{\color{green}{#1}}\def\R#1{\color{red}{#1}}

Table 1. Mean AUROC differences (SHOT − baseline, in percentage points). Green positive values favor SHOT; red negative values favor the baseline.

OOD Dataset / FamilynSHOT- Adv.SHOT - GaussSHOT - Pixel Shuffle
Textures24\G+0.79\G{+0.79}\G+0.56\G{+0.56}R−1.16\R{-1.16}
BROAD-PGD6R−1.26\R{-1.26}R−0.95\R{-0.95}\G+3.28\G{+3.28}
BROAD-AutoAttack6\G+0.06\G{+0.06}\G+0.57\G{+0.57}\G+2.57\G{+2.57}
BROAD-StableDiffussion6R−0.11\R{-0.11}\G+0.31\G{+0.31}\G+0.53\G{+0.53}
BROAD-CondBigGAN6\G+1.20\G{+1.20}\G+0.26\G{+0.26}\G+0.82\G{+0.82}
BROAD-Multilabel6\G+0.81\G{+0.81}\G+2.53\G{+2.53}\G+3.08\G{+3.08}
ImageNet-C6R−0.21\R{-0.21}\G+0.04\G{+0.04}\G+1.76\G{+1.76}
ImageNet-V26\G+0.55\G{+0.55}\G+0.95\G{+0.95}\G+1.34\G{+1.34}
ImageNet-R6R−0.15\R{-0.15}\G+0.41\G{+0.41}\G+1.37\G{+1.37}
iNaturalist12\G+1.57\G{+1.57}\G+1.66\G{+1.66}\G+0.53\G{+0.53}
OpenImage-O12\G+1.23\G{+1.23}\G+1.46\G{+1.46}\G+0.73\G{+0.73}
MNIST12\G+0.41\G{+0.41}\G+6.02\G{+6.02}\G+3.25\G{+3.25}
SVHN12\G+1.06\G{+1.06}\G+4.34\G{+4.34}\G+1.49\G{+1.49}
Places36512\G+0.81\G{+0.81}\G+3.73\G{+3.73}\G+1.37\G{+1.37}
Cross-CIFAR12\G+0.59\G{+0.59}\G+4.79\G{+4.79}\G+2.25\G{+2.25}
Tiny ImageNet12\G+0.57\G{+0.57}\G+4.36\G{+4.36}\G+1.90\G{+1.90}
SSB-hard12\G+1.24\G{+1.24}\G+3.89\G{+3.89}\G+4.25\G{+4.25}
NINCO12\G+1.41\G{+1.41}\G+3.20\G{+3.20}\G+2.67\G{+2.67}
SHOT wins14/1817/1817/18

where n denotes the number of detectors paired with ID dataset/network tested.

4. Computational Cost

We acknowledge SHOT's additional computational cost. However, this is an offline tuning cost, not an added inference-time cost, and it enables tuning without access to real OOD samples.

More importantly, the value of SHOT should not be judged solely by gains on benchmarks. Its central contribution is enabling broadly generalizable detector tuning under a realistic and fundamentally challenging setting where the future OOD distribution is unknown.

Closing Appeal

We respectfully ask the AC and reviewers to evaluate the contribution as a whole: a fundamental practical problem, a new principled framework grounded in approximating the ID/OOD boundary, and extensive empirical evidence, including comparisons against reviewer-suggested alternatives, that supports its generalization across diverse unseen OOD distributions.

We believe this combination constitutes a significant methodological and practical contribution to the OOD detection literature, extending well beyond benchmark improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.