- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the problem of hyperparameter tuning for OOD detectors when no separate OOD tuning dataset is available. The authors argue that existing OOD detection benchmarks often rely on arbitrarily selected OOD validation datasets, which may not represent the unknown distributions encountered during deployment. They first investigate two alternative tuning strategies based on Gaussian noise and adversarial perturbations. To address the limitations of these approaches, the paper proposes Simulated Hold-Out Tuning (SHOT), which repeatedly partitions the available ID classes into held-in and held-out subsets. Surrogate classifiers are trained on the held-in classes, while the held-out classes serve as simulated OOD examples. OOD detector hyperparameters are then selected through Bayesian optimization over these simulated detection tasks.
Critical strengths
- The proposed SHOT framework is conceptually intuitive and detector-agnostic. Using held-out ID classes to construct simulated OOD detection tasks provides a natural way to exploit existing labeled data without collecting external OOD examples.
- SHOT achieves encouraging results, especially for detectors with larger hyperparameter spaces, c.f., Table 3.
- This is a meaningful problem because representative OOD data are often unavailable in real deployments.
Critical weaknesses
- Firstly, high comp cost. The paper reports that training one ImageNet-1K ResNet-50 surrogate takes approximately 20–24 hours using four RTX 3090 GPUs. With 30 surrogate networks in the full configuration, the aggregate computation is substantial.
- The most essential problem of this paper, about its core idea, removing classes from training can substantially change the learned representation, classification head, feature geometry, and confidence distribution. Even when only a small number of classes is removed, there is no general guarantee that the resulting surrogate decision boundary remains close to that of the original network.
- The benchmark and evaluation paradigm are quite old-school. Please consider more modern setting beyond the classification task, which is extensively studied few years ago. Moreover, the selected classification tasks in your main exp are also naive.
- The performance looks incremental and trivial. In several experiments, SHOT improves AUROC by only a small fraction of a percentage point, potentially within the reported standard deviation.
Questions for authors
- Could the authors compare SHOT with computationally inexpensive ID-only tuning methods?
- Could the authors report the average and median AUROC improvements, along with statistical significance, instead of relying primarily on the number of wins?
- Could the authors provide a more detailed end-to-end cost–performance comparison (would be better with the code evidence)?
3 comments
I appreciate the efforts and time from the authors for the additional explaination.
However, considering both novelty and empirical advantage, I'm sorry I can not be more positive, and will keep my current rating.
We would appreciate it if the reviewer could explicitly identify which aspects of our methodological contribution lack novelty, with reference to relevant prior work. Regarding empirical advantage, we ask the reviewer to clarify the scientific basis for deeming our empirical contributions insufficient, particularly in light of the extensive evidence across 180 detector-OOD configurations presented in our global response/final remarks, demonstrating a breadth of generalization across diverse OOD distributions that none of the competing tuning approaches consistently achieves.
W1. High computational cost …
Response: The computational cost is incurred entirely offline, with no additional inference overhead, and surrogate networks can be reused across multiple OOD detectors. Table 5 demonstrates that reducing the number of surrogate networks often achieves comparable AUROC on ImageNet-1K. Furthermore, Appendix G estimates the full SHOT computation at approximately 1,300 USD, substantially below the illustrative 35,000 USD – 50,000 USD cost of collecting OOD data. Importantly, unlike collecting known anomalies, SHOT does not assume access to representative future OOD data.
W2. Even when only a small number of classes is removed, there is no general guarantee that the resulting surrogate decision boundary remains close to that of the original network
Response: There may be a gap between the boundary recovered and the true ID/OOD boundary; SHOT can still be used and the error rate is determined by the gap size. Given that the image statistics match between held out and held in, the gap is generally small (but this can depend on the particular dataset as the reviewer notes). Empirical out-performance of our method over competing methods suggests the gap in SHOT is smaller than competing methods. We will clarify the geometric intuition in Section 4 to avoid implying that an exact boundary approximation is required.
W3. The benchmark and evaluation paradigm are quite old-school. Please consider more modern setting beyond the classification task, which is extensively studied few years ago …
Response: Our focus is on the fundamental problem of OOD detector tuning without given OOD data, rather than developing OOD detectors for specific tasks. Classification provides a standardized and rigorous setting through OpenOOD and BROAD, enabling systematic evaluation across six detector families and diverse semantic and covariate shifts, including ImageNet-1K. Extending SHOT to other tasks would require task-specific adaptations and evaluation protocols, constituting a separate research problem beyond the scope of this paper. The significance of the tuning problem is independent of the complexity of the underlying prediction task.
W4. The performance looks incremental and trivial. In several experiments, SHOT improves AUROC by only a small fraction of a percentage point, potentially within the reported standard deviation.
Response: We disagree that SHOT's improvements are merely incremental or trivial. To quantify their magnitude, we analyzed AUROC gains over the better of Gaussian and adversarial tuning across 180 evaluations. SHOT achieves improvements of at least 1 AUROC point in 36 cases, at least 2 points in 25 cases, and at least 5 points in 5 cases, with a maximum gain of 8.56 points. Thus, improvements are substantial in many settings, even if modest in others. More importantly, SHOT's primary contribution is enabling effective OOD detector tuning without any given OOD data, rather than merely maximizing AUROC gains. These results demonstrate that SHOT can provide meaningful improvements while eliminating reliance on externally supplied OOD tuning data (Table 4).
Q1. Could the authors compare SHOT with computationally inexpensive ID-only tuning methods?
Response: Gaussian and Adv are computationally inexpensive ID only tuning. We additionally compared to inexpensive augmentations of ID data as suggested by Reviewer 6T6c (see Response to W3 of Reviewer 6T6c) - SHOT is better.
Q2. Could the authors report the average and median AUROC improvements, along with statistical significance, instead of relying primarily on the number of wins?
Response: Please see Response to W4.
Q3. Could the authors provide a more detailed end-to-end cost–performance comparison (would be better with the code evidence)?
Response: Please refer to L418-425 of the manuscript (“Computational Cost”). We will release the code after acceptance.