acceptodds
Under review as a conference paper at ICLR 2027

Dataset Identity, Not Novelty : The Source of an Inflated OOD Detection Gain

Abstract

A post-hoc out-of-distribution (OOD) detector reads the activations of a trained classifier and returns a score. It fits that score on in-distribution data, and the benchmarks that evaluate it supply a second piece of OOD data for the fitting itself. Some detectors tune a constant on it. Others fit a direction in feature space or train a flexible combiner and report the number that it reaches as the gain that is still available. Every one of these fits goes through the same check, which validates it on held-out samples of the same OOD dataset. That check rules out memorizing individual images. It says nothing about a fit that has instead learned which dataset it is looking at, and a direction that recognizes one OOD dataset rather than novelty passes it perfectly. The detector that a practitioner installs meets OOD data from a source that nobody fitted it on, so the difference decides what the reported number is worth. We measure it by holding out the whole OOD dataset rather than a sample of it, and we call that gap the inflation. We read it across a range of combiners on ImageNet backbones and CIFAR-100 architectures. Most of the gain that the usual protocol reports turns out to be dataset identity rather than novelty. The size of the fit does not move what survives, so the effect is not ordinary overfitting. The share depends instead on whether the input exposes class identity, and two controls that vary that property alone separate the inflation on every backbone of both benchmarks. A closed form accounts for the effect and computes it from the fitting rows, so a practitioner can tell which fits will inflate without running the hold-out protocol at all. One of these fits survives, namely the single constant that the field already picks on a designated validation dataset. Anything above it reports a gain that the hold-out protocol does not return, and on one benchmark what survives falls while what is reported climbs.

Primary area:
datasets and benchmarks
Keywords:
out-of-distribution detection, benchmark evaluation, model selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WeFiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work addresses a hyper-parameter tuning setting in post-hoc Out-Of-Distribution (OOD) detection methods: hyper-parameters are tuned correspondingly for different OOD datasets. The authors show that the performance gains are derived from the so-called “dataset identity” instead of “novelty”. The authors then devise a metric “inflation” with associated experiments to support this point.

Critical strengths
  1. Experiments cover a variety of backbones on ImageNet-1K and CIFAR100.
  2. The inflation metric is somehow reasonable and interesting.
Critical weaknesses
  1. The paper addresses a tuning protocol that is not representative of common practice. Most post-hoc OOD detectors either require no fitted constant (e.g., MSP, Energy) or select one set of hyperparameters on a validation OOD split and apply it unchanged to all evaluated OOD datasets under the same ID data and backbone (e.g., KNN, ReAct). The paper nonetheless frames target-specific tuning as the main pathology. As a result, the practical severity of the claimed problem is unclear, and the paper may be criticizing a protocol that the field does not actually use. The authors should clarify how prevalent target-specific tuning is in the literature and either justify it as a realistic deployment setting or substantially temper the claims.

  2. The central evidence in Table 1 comes from a self-constructed toy family rather than actual post-hoc OOD methods. The capacity sequence is built from ridge discriminants and small MLPs on a hand-designed channel inventory, whose inputs are abstract scalar statistics computed from penultimate features (e.g., sorted distance profiles, variance of class distances). This proxy family does not capture the structure of actual detectors such as MSP, Energy, KNN, or ReAct, and it is not the object that the paper’s claims target. Moreover, the paper does not run the within/hold-out protocol on representative existing detectors; they appear only as reference baselines. Consequently, the conclusion that “the gain comes from dataset identity” is demonstrated only for the toy proxy, and its generality to real OOD detection methods remains unverified. The authors should either add within/hold-out measurements for representative existing detectors or explicitly narrow the claims to the studied proxy family.

Questions for authors
  1. Could the authors clarify how prevalent target-specific tuning actually is in the post-hoc OOD literature, given that the “validation tuned” setting in Table 1 shows essentially zero inflation?

  2. Could the authors report within/hold-out inflation for representative existing post-hoc OOD detectors (e.g., MSP, Energy, KNN, ReAct), or justify why the self-constructed ridge/MLP proxy on abstract channel statistics is sufficient to represent real OOD detection methods?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the detailed review. Both weaknesses ask whether the protocol and the fits that we measure are the ones the field uses. We answer with the scope of the paper, the literature and measurements on published detectors.


Which protocol the paper addresses (W1). We agree that most post-hoc detectors either fit nothing or pick a single constant on a designated validation split, and the paper does not argue against that practice. It measures it and exempts it. The validation-tuned row of Table 1 inflates by −0.02 on ImageNet and +0.01 on CIFAR-100, and Section 5.3 calls the choice "sound practice at that capacity". The abstract states the same scope: "One of these fits survives, namely the single constant that the field already picks on a designated validation dataset." What the paper measures is a fit above a single constant whose parameters are fitted or validated on samples of the OOD dataset that it is scored on.

How prevalent that protocol is (W1). It is common in the literature, including at the main venues.

  • Methods whose own protocol fits on samples of each evaluated OOD dataset: Lee et al. (NeurIPS 2018) tune their layer weights and input perturbation on 1,000 samples per OOD dataset, and Yu & Aizawa (ICCV 2019), Zisselman & Tamar (CVPR 2020) and Gomes et al. (ICLR 2022) keep that protocol. Dong et al. (CVPR 2022) train their detector on ID and OOD samples. Sehwag et al. (ICLR 2021) and Fort et al. (NeurIPS 2021) fit on a few images per class of the targeted OOD dataset. Xu et al. (NeurIPS 2023) fit on 1% of the evaluated ID and OOD data and present the near-perfect result as "the strong potential of our method". Novello et al. (2024) select which scores to combine on one half of the OOD test data.
  • Comparisons that tune the baselines on each OOD dataset: Hsu et al. (CVPR 2020), Sastry & Oore (ICML 2020) and Tajwar et al. (2021).

Two of these papers describe the protocol as the norm. Dong et al. write that "Conventional OOD detection approaches assume the access to both ID and OOD data for OOD detector training and hyper-parameter tuning", and Hsu et al. write that "many recent methods based on neural networks rely on training or tuning with both in-distribution and out-of-distribution data". We agree that the submission did not show how common this protocol is, and we restructured the Related Work to make it explicit. Its second paragraph now sorts fits by where their OOD data comes from, a designated validation split or samples of the OOD datasets under evaluation, and collects the cases above.

Deployment (W1). We do not justify this protocol as a deployment setting, because the paper's point is that it is not one. A deployed detector meets OOD data from sources that nobody fitted it on. The hold-out protocol therefore gives the number that deployment can expect, and the inflation is the part of the reported gain that deployment does not get. Some of the works above argue that a few known outliers can be collected in practice (Fort et al.; Sehwag et al.). Even then the detector meets OOD data beyond that source, and the inflation measures how much of the reported gain stays with that source. Dong et al. draw the same line when they add a protocol that trains on one OOD dataset and evaluates on unseen ones. The scope of the claim already excludes the exempt case, so we keep the claim as the abstract states it.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Published detectors (W2). Table 1 already runs a published detector under both protocols. Its two single-constant rows tune the constant α of MahaVar (Kim & Yoon, 2026), and the validation-tuned row picks α on the OpenOOD validation split, as that paper does. When α is tuned on the target dataset, the inflation is +0.52 on ImageNet and +1.53 on CIFAR-100. When α is picked on the validation split, it is −0.02 and +0.01.

The steps above it have the structure of published fits. The channels are existing detector scores: channel 1 is the Mahalanobis++ score, channel 6 is the variance term of MahaVar and channel 7 is the Energy score. Channels 1 and 7 match the AUROC of those detectors in our detector table to within 0.0006 on every backbone of both benchmarks. Combining such scores is what Novello et al. do. A logistic regression on layer-wise scores is what Lee et al. and Zisselman & Tamar fit, and a linear or MLP classifier on features is what Fort et al. and Dong et al. train. The capacity sequence places these forms on one axis with everything else fixed, which lets the paper separate the parameter count from the input. Published detectors change both at once.

We also ran two published fits of this kind under both protocols. The gain that the pair selection of Novello et al. reports over the best single score is +1.26 on ImageNet and +0.83 on CIFAR-100, and it falls to −0.42 and −0.48 when the target dataset is held out. The gain that the layer ensemble of Lee et al. reports over its own last layer is +2.30 and +8.09, and it falls to −0.35 and +2.06. Lee et al. run on the two ImageNet backbones for which we cache layer features, and the pair selection draws from our 13 channels.

Q1. The near-zero inflation of the validation-tuned row is the exempt case above, a single constant picked on a designated validation split. The prevalence question concerns the fits above it, and the list under W1 answers it.

Q2. MSP, Energy and MaxLogit fit no parameter, so both protocols give the same number and the inflation is zero by construction. KNN and ReAct each have one constant and behave as the single-constant row predicts. When the constant is tuned on the target dataset, the inflation is +1.05 / +1.99 for KNN and +0.09 / +1.70 for ReAct on ImageNet / CIFAR-100. Adding the sparsity of DICE to ReAct as a second constant raises the inflation to +1.08 / +2.74. When the constants are picked on the designated validation dataset, the inflation of all three is at most 0.32.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

References cited in our reply

  • Dong et al. (2022). Neural Mean Discrepancy for Efficient Out-of-Distribution Detection. CVPR.
  • Fort et al. (2021). Exploring the Limits of Out-of-Distribution Detection. NeurIPS.
  • Gomes et al. (2022). Igeood: An Information Geometry Approach to Out-of-Distribution Detection. ICLR.
  • Hendrycks & Gimpel (2017). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. ICLR.
  • Hendrycks et al. (2022). Scaling Out-of-Distribution Detection for Real-World Settings. ICML.
  • Hsu et al. (2020). Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data. CVPR.
  • Kim & Yoon (2026). MahaVar: OOD Detection via Class-wise Mahalanobis Distance Variance under Neural Collapse. NeurIPS.
  • Lee et al. (2018). A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NeurIPS.
  • Liu et al. (2020). Energy-based Out-of-distribution Detection. NeurIPS.
  • Müller & Hein (2025). Mahalanobis++: Improving OOD Detection via Feature Normalization. ICML.
  • Novello et al. (2024). Improving Out-of-Distribution Detection by Combining Existing Post-hoc Methods. arXiv:2407.07135.
  • Sastry & Oore (2020). Detecting Out-of-Distribution Examples with Gram Matrices. ICML.
  • Sehwag et al. (2021). SSD: A Unified Framework for Self-Supervised Outlier Detection. ICLR.
  • Sun & Li (2022). DICE: Leveraging Sparsification for Out-of-Distribution Detection. ECCV.
  • Sun et al. (2021). ReAct: Out-of-distribution Detection With Rectified Activations. NeurIPS.
  • Sun et al. (2022). Out-of-Distribution Detection with Deep Nearest Neighbors. ICML.
  • Tajwar et al. (2021). No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets. ICML Workshop on Uncertainty and Robustness in Deep Learning.
  • Xu et al. (2023). VRA: Variational Rectified Activation for Out-of-distribution Detection. NeurIPS.
  • Yu & Aizawa (2019). Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancy. ICCV.
  • Zhang et al. (2024). OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. Journal of Data-centric Machine Learning Research.
  • Zisselman & Tamar (2020). Deep Residual Flow for Out of Distribution Detection. CVPR.
Reviewer_jyaBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper constructs OOD (out-of-distribution) representations based on classifier features by applying normalization, whitening, and class-distance calculations, and subsequently trains constant, Ridge, and MLP-based detectors. By comparing a "within" protocol (where the target dataset is used during model fitting) against a "hold-out" protocol (where the target dataset is entirely withheld), the authors quantify the performance inflation attributable to "dataset identity." Furthermore, they analyze the origins of this phenomenon through experiments involving ranking distances and random projections.

Critical strengths
  1. The paper addresses an important issue in OOD evaluation: tuning on a particular OOD dataset may produce overly optimistic results. The within, hold-out, and inflation definitions are clear, and the experiments show that within performance can improve without comparable cross-dataset gains.

  2. Within the selected ImageNet-1k and CIFAR-100 benchmarks, the paper covers multiple backbones, OOD datasets, and input representations.

Critical weaknesses
  1. The paper cites the OD-test, Tajwar et al., and domain generalization, but does not clearly distinguish its contribution from these works. It should also discuss SC-OOD. In particular, the paper should clarify whether its main novelty is the inflation measure, the analysis of dataset-specific signals, or something else. Leave-one-dataset-out evaluation itself is not new.

  2. The experiments use only ImageNet-1k, CIFAR-100, and mostly supervised classification models. They do not test modern vision foundation models or application-specific datasets. This limits the deployment-related claims. The authors need not cover every model and domain, but they should add some representative modern models.

  3. The within and hold-out protocols differ in more than whether the target dataset is observed. They also use different numbers of fitting samples, dataset weights, and data mixtures. The current analyses do not fully remove these confounds. Matched-size, equal-weight, and repeated-split controls would make the inflation measurement more convincing. In addition, lower hold-out performance shows poor cross-dataset transfer, but does not prove that the within-dataset signal is invalid dataset identity; it may include legitimate target-specific novelty. The sorted/unsorted comparison also changes the geometry seen by the linear model, so it supports the role of class identity without isolating it completely.

  4. The closed-form analysis applies to linear discriminants with shared covariance and does not provide a general explanation for MLPs or other nonlinear detectors. The empirical study also uses limited detector types, two benchmarks, and fixed splits. Although the paper acknowledges these limitations, that does not resolve their effect on the main claims. The authors should either add repeated-split or uncertainty analyses and broader detectors.

Questions for authors

see "Critical Weaknesses"

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the careful review and the concrete list of controls. Every control that the review lists is now in the revised paper, and we answer in the order of the review.


Novelty and prior work (W1). We do not claim leave-one-dataset-out evaluation. Section 4 says so: "Cross-validation that leaves out one dataset at a time is a known idea and we do not claim it." Nor is the observation that fits transfer poorly new. Our contribution is to measure why. It has three parts, which the revised introduction now states as its contributions.

  • We split a reported gain in two on the same evaluation rows. One part transfers to a different OOD dataset and the other does not, and both are read against a zero point that has no constant to tune.
  • We find what sets the split. The parameter count barely moves it: a network on the same 13 channels grows from 13 to 5.4×1055.4 × 10^5 parameters while its held-out arm moves by −0.34 (Appendix K). Whether the input exposes class identity does move it. Two controls that hold dimension and parameter count fixed separate the inflation on every backbone of both benchmarks.
  • We derive a closed form, the length of the target's mean shift times 1−cos⁡θt,1 − \cos \theta_t, which predicts the inflation from the fitting rows alone (rank correlation 0.973 over 274 pairs on ImageNet).

The relation to the works that the review names is now in the third paragraph of the revised Related Work. Shafaei et al. (OD-test) observe that detectors with more degrees of freedom overestimate their accuracy more. Our measurement splits those degrees of freedom into parameter count and input, and only the input matters much. Tajwar et al. report that no detector leads across dataset pairs, which is a symptom of the effect that we measure. The rule itself comes from domain generalization, where Gulrajani & Lopez-Paz call test-domain selection oracle selection, and we credit it there. Yang et al. (SC-OOD) argue that a split by data source lets detectors succeed on low-level differences between sources, and they relabel the benchmark. We keep the labels of the benchmark and measure how much of a fitted gain is dataset identity, how that share depends on capacity and on the input, and what sets it.

Modern models (W2). We added three foundation-model classifiers, one for each way that such a model becomes an ImageNet-1k classifier: CLIP ViT-B/16 zero-shot, DINOv2 ViT-B/14 with its official linear head, and CLIP ViT-B/16 pretrained on LAION-2B and fine-tuned on ImageNet, the checkpoint that Mahalanobis++ evaluates. Their top-1 accuracies (68.32, 84.53, 86.20) match the published values, and our Mahalanobis++ AUROC on the last one is 93.12 against the reported 93.3. The results are in the new Appendix P (Table 34).

CLIP zero-shotDINOv2 linear probeCLIP fine-tuned
sorted / unsorted profile (13 dims)+6.79 / +22.50+2.78 / +8.10+1.34 / +2.45
13 channels / 13 random projections+7.15 / +19.77+1.54 / +9.95+1.21 / +11.09
validation-tuned single constant+0.13+0.15+0.02
rank correlation of the closed form0.9731.0000.964

On each model the class-aware member of both controls inflates more, the validation-tuned constant inflates by close to nothing, and the closed form ranks the measured gaps. On the 13 channels zero-shot CLIP inflates more than any of the 11 backbones of Table 1 (+7.15, against at most +3.93 and +1.94 on average). The original 11 backbones also include large-scale pretraining (ViT-B/16 SWAG, ConvNeXtV2-B with FCMAE and ImageNet-22k, SwinV2-B and DeiT3-B/16 on ImageNet-22k). We did not run application-specific benchmarks in this revision.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Confounds (W3). The new Appendix Q (Table 35) recomputes the main rows of Table 1 under the controls that the review lists.

  • Repeated splits: over ten random splits of the ID and OOD halves, the standard deviation of the benchmark average is at most 0.09 on ImageNet and 0.29 on CIFAR-100 (0.35 and 0.73 for the principal-component row). Every row of Table 35 is positive on every backbone in every split.
  • Matched size and equal weight: when both arms fit on the same number of OOD rows, with every other dataset contributing an equal share, every row stays positive on every backbone. The sizes change, and every sign stays the same.
  • Equal weight alone was already in the submission for the permutation pair (Table 6).

Legitimate target-specific signal (W3). We agree that a lower held-out score shows poor transfer and does not by itself say what the within-dataset signal is. Three results narrow it down.

  • The first control holds content, dimension and parameter count fixed and varies only whether the input reveals which class a sample is near. That alone multiplies the inflation by three to six. The part that does not transfer follows where the target dataset sits among the classes, which is a property of the dataset.
  • In the closed form, the inflation is the part of the target's mean shift that the other OOD datasets do not share.
  • Whatever that part is called, a deployed detector cannot use it, because deployment does not know its target in advance. If the target is known, as with a few collected outliers, the signal is usable, but the task is then the detection of that known source rather than OOD detection in general. We call the part dataset identity because it is tied to the dataset, not because it carries no information.

Geometry of the sorted / unsorted comparison (W3). The new Section 5.7 splits the predicted gap between the two members into a length term and an angle term, and Appendix R (Table 36) gives the values. At 13 directions the length term is negative, because the sorted member has the longer mean shift and separates OOD from ID better. The angle term is larger on every one of the 17 backbones (ImageNet +0.167 = −0.206 + 0.373, CIFAR-100 +0.419 = −0.089 + 0.508, on the binormal scale). The pair of 13 channels and 13 random projections involves no sorting and splits the same way. A network, which is not limited to the geometry that a linear model sees, keeps the ordering too (see W4 below).

Closed form, detectors, benchmarks and splits (W4).

  • Closed form: it is a derivation for a linear score, and we keep its scope as Section 6 states it. The main claims do not rest on it, because they are measured directly. Table 1 now also fits the network on both controls (sorted and unsorted profile, 13 random projections). With the network the class-aware member still inflates more on every backbone of both benchmarks (unsorted − sorted +9.62 / +7.29; random projections − channels +8.88 / +10.56).
  • Detector types: Table 1 already contains a published detector under both protocols. Its single-constant rows tune the constant α of MahaVar (Kim & Yoon, 2026), and its channels include the Mahalanobis++ and Energy scores. We also ran two published multi-parameter fits under both protocols. The gain that the pair selection of Novello et al. reports over the best single score is +1.26 on ImageNet and +0.83 on CIFAR-100 and falls to −0.42 and −0.48 when held out. The gain of the layer ensemble of Lee et al. over its own last layer is +2.30 and +8.09 and falls to −0.35 and +2.06.
  • Fixed splits: besides Table 35, the two published fits keep their inflation over the same ten splits (Novello et al. +2.34 ± 0.06 / +4.08 ± 0.12, Lee et al. +2.63 ± 0.10 / +6.20 ± 0.21).
  • Benchmarks: a third ID benchmark is not in this revision, as the Limitations state. The model axis is wider. The public OpenOOD CIFAR-100 checkpoints reproduce every conclusion of Table 1 through the same pipeline (see our reply to Reviewer 4XUT), and the three foundation models above add a different training regime.
AuthorsOpenReview ↗

Official Comment by Authors (3/3)

References cited in our reply

  • Gulrajani & Lopez-Paz (2021). In Search of Lost Domain Generalization. ICLR.
  • Kim & Yoon (2026). MahaVar: OOD Detection via Class-wise Mahalanobis Distance Variance under Neural Collapse. NeurIPS.
  • Lee et al. (2018). A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. NeurIPS.
  • Müller & Hein (2025). Mahalanobis++: Improving OOD Detection via Feature Normalization. ICML.
  • Novello et al. (2024). Improving Out-of-Distribution Detection by Combining Existing Post-hoc Methods. arXiv:2407.07135.
  • Oquab et al. (2024). DINOv2: Learning Robust Visual Features without Supervision. TMLR.
  • Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML.
  • Shafaei et al. (2019). A Less Biased Evaluation of Out-of-distribution Sample Detectors. British Machine Vision Conference.
  • Tajwar et al. (2021). No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets. ICML Workshop on Uncertainty and Robustness in Deep Learning.
  • Yang et al. (2021). Semantically Coherent Out-of-Distribution Detection. ICCV.
  • Zhang et al. (2024). OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. Journal of Data-centric Machine Learning Research.
AuthorsOpenReview ↗

Official Comment by Authors - 2

We have posted our response along with a revised version of the paper. We'd be glad to hear whether it addresses your concerns. If anything is still unclear, we're happy to clarify before the deadline. The controls from your list are now in the revised paper (Table 1 and Appendices P–R).

Thanks again for your review.

Reviewer_4XUTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the problem where an OOD detector learns to recognize OOD samples not because it learns a generalizable rule, but because the detector recognizes the dataset. This problem is quantified by using the gap between the within protocol and the held-out protocol, where the first protocol trains the detector on one half of the OOD dataset to be tested, whereas the second protocol does not use that half. A geometrical interpretation of the gap is provided.

Critical strengths
  1. The experiment is comprehensive, with 33 tables in total, clearly demonstrating the gap and many related experiments.
  2. The geometric interpretation of the gap is insightful and intuitive.
Critical weaknesses
  1. Line 233 claims “CIFAR-100 has no public checkpoint of this kind”, and Line 689 narrows this claim to “No pretrained CIFAR-100 checkpoint exists in timm.” However, the public checkpoints are available at ​​https://github.com/Jingkang50/OpenOOD#pre-trained-checkpoints. This raises the concern whether the reported results are comparable to existing results and weakens the reproducibility.
  2. The writing needs significant improvement. The writing is not standard in numerous places. A very short list of such cases are:
  • The word “material” is defined much later but already used in the introduction.
  • In line 195, a hyphen should be added in “target tuned”. There are many other similar places throughout the paper where hyphens should be added.
  • Many verb uses are understandable but are confusing under the technical context. One example is the section title “what a fit buys and what survives”. There are straightforward technical terms in the existing literature of OOD detection. I will consider raising my score if these weaknesses can be adequately addressed.
Questions for authors
  1. Could you clarify why publicly available checkpoints for CIFAR-100 are not used in this paper? Since you used the models that you train yourself, could you demonstrate that your trained models are comparable to public checkpoints?
  2. In Table 1, on CIFAR-100, the “held out” column shows that most of the methods underperformed VarRatio that require no fitting at all. The trend is very different from the ImageNet-1k results on the left, where many methods outperformed VarRatio. Could you provide any insights?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the careful reading and the concrete examples. We answer in the order of the review.


Public CIFAR-100 checkpoints (W1, Q1). OpenOOD v1.5 releases CIFAR-100 classifiers for a single architecture, ResNet-18 (three training runs). The other five architectures of our study (ResNet-34, WRN-28-10, DenseNet-BC, VGG16-BN and MobileNetV2) have no OpenOOD checkpoint. We train all six under one recipe so that the recipe never varies with the architecture. We revised the sentences at lines 233 and 689, which now state what we train (line numbers refer to the submitted version).

To show that our models are comparable, we ran the three OpenOOD ResNet-18 checkpoints (best.ckpt, the file that the OpenOOD evaluator loads) through our pipeline unchanged. The data, preprocessing, splits and measurement code are the same, and only the model differs. Values are means over the three runs.

OpenOOD ResNet-18our ResNet-18
top-1 accuracy77.2978.08
VarRatio AUROC (highest detector on both)81.3881.61
inflation, rows of Table 1
single constant, target-tuned+0.92+0.75
2 / 4 / 8 / 13 channels+1.96 / +2.51 / +4.33 / +4.60+1.77 / +2.07 / +3.41 / +3.64
{d(1),Var⁡}\{d_{(1)}, \operatorname{Var}\} ridge+0.89+0.84
13 channels, MLP+6.54+6.72
zz, ridge / zz, MLP+18.52 / +20.20+15.26 / +17.29
sorted / unsorted profile+4.26 / +21.86+4.62 / +21.77
13 channels / 13 random projections of zz / 13 leading principal components of zz+4.60 / +18.10 / +16.78+3.64 / +19.41 / +19.31
sorted / unsorted profile, MLP+6.08 / +16.31+6.82 / +15.02
13 random projections of zz, MLP+19.37+18.08
single constant, validation-tuned+0.19+0.20

On the same checkpoints MSP, MaxLogit and Energy come within about one AUROC point of the OpenOOD v1.5 leaderboard (CIFAR-100, ResNet-18), at −1.04, −1.07 and −1.06 on near-OOD and −0.96, −0.73 and −0.69 on far-OOD. The offset is uniform across the three scores and is consistent with the three data differences listed in Appendix A and with OpenOOD's 9,000-image ID test split.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Writing (W2). We revised the wording throughout the paper, including the appendix and the figures. The changes below are representative, with line numbers of the submitted version.

  • "Material" before its definition. The introduction no longer uses the word. Line 111 now reads "the parameter count moves the split far less than the choice of input does". The first use of "material" is its definition in Section 3.

  • Hyphens. Line 195 now reads "we call it target-tuned". Compound modifiers carry the hyphen throughout, for example "validation-tuned", "held-out", "class-aware" and "permutation-invariant", including the table headers and the figure legends.

  • Verbs. We replaced the figurative verbs with plain ones. (Line numbers refer to the submitted version)

LineSubmittedRevised
173 (Sec. 4 title)Measuring what a fit buysMeasuring the inflation
290 (Sec. 5.3 title)What a fit buys and what survivesDataset identity in the reported gain
321 (Sec. 5.4 title)Class identity is what the fit buysExposing class identity raises the inflation
365 (Sec. 5.5 title)The material is what sets the inflationThe material sets the inflation
293CIFAR-100 pays more than ImageNet at every stepCIFAR-100 inflates more than ImageNet at every step
306One step escapes the patternOne step is an exception to the pattern
404The within protocol hands the fit w=Σ−1Δtw = \Sigma^{-1}\Delta_tThe within protocol gives the fit w=Σ−1Δtw = \Sigma^{-1}\Delta_t
  • Technical terms. Where a standard term means the same thing, the paper now uses it.
LineSubmittedRevised
151Two readouts of itTwo statistics of it
257random orthonormal views of the whitened featurerandom orthonormal projections of the whitened feature
402the displacement of OOD set ttthe mean shift of OOD set tt
Table 113 top PCA views of zz13 leading principal components of zz

Q1. Answered under W1 above.

Q2 (CIFAR-100 held-out column). The closed form of Section 5.6 explains the difference. The gap between the two protocols is the length of the target's mean shift times 1−cos⁡θt1 - \cos\theta_t, where θt\theta_t is the angle between that mean shift and the average mean shift of the other OOD datasets. On CIFAR-100 the other datasets share much less of each target's mean shift. Averaged over every representation, the cosine is 0.817 for the permutation-invariant material and 0.208 for the class-aware material, against 0.910 and 0.418 on ImageNet. A direction fitted on the other datasets therefore transfers less to the target, and the held-out AUROC falls below the zero-capacity reference. The closed form still ranks the measured gaps on CIFAR-100, with a rank correlation of 0.978 over 146 (representation, architecture) pairs. We added this explanation to Section 5.7.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

References cited in our reply

  • Hendrycks & Gimpel (2017). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. ICLR.
  • Hendrycks et al. (2022). Scaling Out-of-Distribution Detection for Real-World Settings. ICML.
  • Liu et al. (2020). Energy-based Out-of-distribution Detection. NeurIPS.
  • Zhang et al. (2024). OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. Journal of Data-centric Machine Learning Research.
AuthorsOpenReview ↗

Official Comment by Authors - 2

We have posted our response along with a revised version of the paper. We'd be glad to hear whether it addresses your concerns. If anything is still unclear, we're happy to clarify before the deadline. Every change is marked in blue in the revised PDF.

Thanks again for your review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response by Authors

We sincerely thank all reviewers for their careful reading and for comments that made the paper clearer and its evidence broader. We have uploaded a revised version. Every change against the submitted version is shown in blue: changed words in blue within otherwise unchanged sentences, and new sentences, table rows and appendix sections in blue throughout.

Summary of the revision

  • Writing. We revised the wording throughout the paper, appendix and figures: hyphens on compound modifiers, plain verbs in place of figurative ones, four section titles, and standard terms where they mean exactly the same thing (random projections, leading principal components, mean shift, statistic). The word "material" is no longer used before its definition.
  • Related work. The second paragraph now sorts fits by where their OOD data comes from, a designated validation split or samples of the OOD datasets under evaluation, and collects published cases of the latter. A new third paragraph relates the paper to prior observations that fitted gains do not transfer (OD-test, Tajwar et al., domain generalization, SC-OOD) and states our position. The contributions in the introduction are stated directly.
  • Nonlinear fits. Table 1 now also fits the network on both class-identity controls, and the ordering holds on every backbone of both benchmarks.
  • The geometry of the controls. A new Section 5.7 splits the predicted gap of the sorted / unsorted control exactly into a length term and an angle term. At matched width the angle term is the larger of the two on all 17 backbones.
  • Modern models. New Appendix P repeats the measurement on three foundation-model classifiers (CLIP zero-shot, DINOv2 with its linear head, CLIP fine-tuned on ImageNet).
  • Confounds and uncertainty. New Appendix Q repeats the main rows of Table 1 over ten random splits and with fitting pools matched in size and in per-dataset weight.
  • CIFAR-100 versus ImageNet. Section 5.7 adds the cosines that explain why the held-out column falls below the zero-capacity reference on CIFAR-100.
  • Checkpoints. The setup sentence on CIFAR-100 checkpoints now states what we train.

Our replies add measurements that are not in the paper: the public OpenOOD CIFAR-100 checkpoints run through our pipeline, the constants of published detectors and two published multi-parameter procedures under both protocols, and their behavior over repeated splits.

We would welcome any further questions or comments, and we are glad to run additional analyses that would help the assessment during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.