Tuning OOD Detectors Without Given OOD Data

Many existing out-of-distribution (OOD) detectors are tuned using a separate dataset deemed OOD with respect to the training distribution, often ad hocly chosen. In practical settings, such OOD data may be unavailable or difficult to obtain; even when available, they may not represent actual OOD data, which often include "unknown unknowns.” As such, there is significant variance in detector performance depending on the ad hoc OOD dataset used for tuning. Hence, we introduce and formalize the often neglected problem of tuning OOD detectors without a "given” OOD dataset. We propose and evaluate two strong baseline methods based on related literature and found that they too yield inconsistent performance across detectors and datasets. We address this by introducing simulated hold-out tuning (SHOT), a generic approach that requires no additional data other than the ID training set. SHOT repeatedly holds out subsets of ID classes and treats them as simulated OOD samples for detector tuning. We evaluate these approaches on six OOD detectors across the OpenOOD and BROAD datasets, spanning both semantic and covariate shifts. SHOT achieves the highest number of wins across datasets and detectors compared with the baselines, particularly excelling at higher parameter detectors. These results show that effective OOD detector tuning need not rely on collecting a separate OOD dataset, removing a significant practical barrier to OOD detection in real-world deployments.