- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work addresses a hyper-parameter tuning setting in post-hoc Out-Of-Distribution (OOD) detection methods: hyper-parameters are tuned correspondingly for different OOD datasets. The authors show that the performance gains are derived from the so-called “dataset identity” instead of “novelty”. The authors then devise a metric “inflation” with associated experiments to support this point.
Critical strengths
- Experiments cover a variety of backbones on ImageNet-1K and CIFAR100.
- The inflation metric is somehow reasonable and interesting.
Critical weaknesses
-
The paper addresses a tuning protocol that is not representative of common practice. Most post-hoc OOD detectors either require no fitted constant (e.g., MSP, Energy) or select one set of hyperparameters on a validation OOD split and apply it unchanged to all evaluated OOD datasets under the same ID data and backbone (e.g., KNN, ReAct). The paper nonetheless frames target-specific tuning as the main pathology. As a result, the practical severity of the claimed problem is unclear, and the paper may be criticizing a protocol that the field does not actually use. The authors should clarify how prevalent target-specific tuning is in the literature and either justify it as a realistic deployment setting or substantially temper the claims.
-
The central evidence in Table 1 comes from a self-constructed toy family rather than actual post-hoc OOD methods. The capacity sequence is built from ridge discriminants and small MLPs on a hand-designed channel inventory, whose inputs are abstract scalar statistics computed from penultimate features (e.g., sorted distance profiles, variance of class distances). This proxy family does not capture the structure of actual detectors such as MSP, Energy, KNN, or ReAct, and it is not the object that the paper’s claims target. Moreover, the paper does not run the within/hold-out protocol on representative existing detectors; they appear only as reference baselines. Consequently, the conclusion that “the gain comes from dataset identity” is demonstrated only for the toy proxy, and its generality to real OOD detection methods remains unverified. The authors should either add within/hold-out measurements for representative existing detectors or explicitly narrow the claims to the studied proxy family.
Questions for authors
-
Could the authors clarify how prevalent target-specific tuning actually is in the post-hoc OOD literature, given that the “validation tuned” setting in Table 1 shows essentially zero inflation?
-
Could the authors report within/hold-out inflation for representative existing post-hoc OOD detectors (e.g., MSP, Energy, KNN, ReAct), or justify why the self-constructed ridge/MLP proxy on abstract channel statistics is sufficient to represent real OOD detection methods?
5 comments
Official Comment by Authors (2/3)
Published detectors (W2). Table 1 already runs a published detector under both protocols. Its two single-constant rows tune the constant α of MahaVar (Kim & Yoon, 2026), and the validation-tuned row picks α on the OpenOOD validation split, as that paper does. When α is tuned on the target dataset, the inflation is +0.52 on ImageNet and +1.53 on CIFAR-100. When α is picked on the validation split, it is −0.02 and +0.01.
The steps above it have the structure of published fits. The channels are existing detector scores: channel 1 is the Mahalanobis++ score, channel 6 is the variance term of MahaVar and channel 7 is the Energy score. Channels 1 and 7 match the AUROC of those detectors in our detector table to within 0.0006 on every backbone of both benchmarks. Combining such scores is what Novello et al. do. A logistic regression on layer-wise scores is what Lee et al. and Zisselman & Tamar fit, and a linear or MLP classifier on features is what Fort et al. and Dong et al. train. The capacity sequence places these forms on one axis with everything else fixed, which lets the paper separate the parameter count from the input. Published detectors change both at once.
We also ran two published fits of this kind under both protocols. The gain that the pair selection of Novello et al. reports over the best single score is +1.26 on ImageNet and +0.83 on CIFAR-100, and it falls to −0.42 and −0.48 when the target dataset is held out. The gain that the layer ensemble of Lee et al. reports over its own last layer is +2.30 and +8.09, and it falls to −0.35 and +2.06. Lee et al. run on the two ImageNet backbones for which we cache layer features, and the pair selection draws from our 13 channels.
Q1. The near-zero inflation of the validation-tuned row is the exempt case above, a single constant picked on a designated validation split. The prevalence question concerns the fits above it, and the list under W1 answers it.
Q2. MSP, Energy and MaxLogit fit no parameter, so both protocols give the same number and the inflation is zero by construction. KNN and ReAct each have one constant and behave as the single-constant row predicts. When the constant is tuned on the target dataset, the inflation is +1.05 / +1.99 for KNN and +0.09 / +1.70 for ReAct on ImageNet / CIFAR-100. Adding the sparsity of DICE to ReAct as a second constant raises the inflation to +1.08 / +2.74. When the constants are picked on the designated validation dataset, the inflation of all three is at most 0.32.
Official Comment by Authors (3/3)
References cited in our reply
- Dong et al. (2022). Neural Mean Discrepancy for Efficient Out-of-Distribution Detection. CVPR.
- Fort et al. (2021). Exploring the Limits of Out-of-Distribution Detection. NeurIPS.
- Gomes et al. (2022). Igeood: An Information Geometry Approach to Out-of-Distribution Detection. ICLR.
- Hendrycks & Gimpel (2017). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. ICLR.
- Hendrycks et al. (2022). Scaling Out-of-Distribution Detection for Real-World Settings. ICML.
- Hsu et al. (2020). Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data. CVPR.
- Kim & Yoon (2026). MahaVar: OOD Detection via Class-wise Mahalanobis Distance Variance under Neural Collapse. NeurIPS.
- Lee et al. (2018). A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NeurIPS.
- Liu et al. (2020). Energy-based Out-of-distribution Detection. NeurIPS.
- Müller & Hein (2025). Mahalanobis++: Improving OOD Detection via Feature Normalization. ICML.
- Novello et al. (2024). Improving Out-of-Distribution Detection by Combining Existing Post-hoc Methods. arXiv:2407.07135.
- Sastry & Oore (2020). Detecting Out-of-Distribution Examples with Gram Matrices. ICML.
- Sehwag et al. (2021). SSD: A Unified Framework for Self-Supervised Outlier Detection. ICLR.
- Sun & Li (2022). DICE: Leveraging Sparsification for Out-of-Distribution Detection. ECCV.
- Sun et al. (2021). ReAct: Out-of-distribution Detection With Rectified Activations. NeurIPS.
- Sun et al. (2022). Out-of-Distribution Detection with Deep Nearest Neighbors. ICML.
- Tajwar et al. (2021). No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets. ICML Workshop on Uncertainty and Robustness in Deep Learning.
- Xu et al. (2023). VRA: Variational Rectified Activation for Out-of-distribution Detection. NeurIPS.
- Yu & Aizawa (2019). Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancy. ICCV.
- Zhang et al. (2024). OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. Journal of Data-centric Machine Learning Research.
- Zisselman & Tamar (2020). Deep Residual Flow for Out of Distribution Detection. CVPR.
Official Comment by Authors (1/3)
We thank the reviewer for the detailed review. Both weaknesses ask whether the protocol and the fits that we measure are the ones the field uses. We answer with the scope of the paper, the literature and measurements on published detectors.
Which protocol the paper addresses (W1). We agree that most post-hoc detectors either fit nothing or pick a single constant on a designated validation split, and the paper does not argue against that practice. It measures it and exempts it. The validation-tuned row of Table 1 inflates by −0.02 on ImageNet and +0.01 on CIFAR-100, and Section 5.3 calls the choice "sound practice at that capacity". The abstract states the same scope: "One of these fits survives, namely the single constant that the field already picks on a designated validation dataset." What the paper measures is a fit above a single constant whose parameters are fitted or validated on samples of the OOD dataset that it is scored on.
How prevalent that protocol is (W1). It is common in the literature, including at the main venues.
Two of these papers describe the protocol as the norm. Dong et al. write that "Conventional OOD detection approaches assume the access to both ID and OOD data for OOD detector training and hyper-parameter tuning", and Hsu et al. write that "many recent methods based on neural networks rely on training or tuning with both in-distribution and out-of-distribution data". We agree that the submission did not show how common this protocol is, and we restructured the Related Work to make it explicit. Its second paragraph now sorts fits by where their OOD data comes from, a designated validation split or samples of the OOD datasets under evaluation, and collects the cases above.
Deployment (W1). We do not justify this protocol as a deployment setting, because the paper's point is that it is not one. A deployed detector meets OOD data from sources that nobody fitted it on. The hold-out protocol therefore gives the number that deployment can expect, and the inflation is the part of the reported gain that deployment does not get. Some of the works above argue that a few known outliers can be collected in practice (Fort et al.; Sehwag et al.). Even then the detector meets OOD data beyond that source, and the inflation measures how much of the reported gain stays with that source. Dong et al. draw the same line when they add a protocol that trains on one OOD dataset and evaluates on unseen ones. The scope of the claim already excludes the exempt case, so we keep the claim as the abstract states it.