acceptodds
Withdrawn from ICLR 2027

When OOD Classes Were Seen Upstream: Effects of Class-Specific Supervision on Downstream OOD Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Out-of-distribution (OOD) detection is typically evaluated with respect to a downstream task, but the representation used by the detector may have encountered an OOD class during upstream training. We ask whether the upstream supervision history of a class affects its detectability as OOD downstream. To estimate this effect, we introduce a controlled evaluation in which the downstream task, evaluation data, architecture, training objective, and number of upstream classes are held fixed, while future-OOD classes either receive supervised treatment upstream or are withheld from upstream training. We make this comparison through a grouped rotating leave-one-out construction across multiple semantic classes. Experiments on a WordNet-structured ImageNet subset, CIFAR-100, and iNaturalist 2021 show that, under cosine kNN, future-OOD classes supervised upstream are systematically easier to detect downstream than matched classes withheld from upstream training. The mean AUROC difference is 5.9–7.5 points across the three datasets, and the effect persists with from-scratch Vision Transformers, with differences of 2.6–4.9 points. Across nine established OOD detectors, upstream supervision consistently affects OOD performance, but the magnitude and direction vary by detector. These results show that the upstream provenance of evaluated OOD classes is a previously underexamined source of variation in OOD detection, highlighting the importance of accounting for class provenance in OOD evaluation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
OOD, Classification, Pre-trained Representation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_A5PVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the effect of upstream pre-training on OOD detection. The authors design an experiment, where the model is pre-trained on the given ID data and several sets of OOD classes, while a separate set of OOD classes is preserved and not used in upstream pre-training. Then, they evaluate the model using the ID data and both OOD sets, those the model has observed in upstream pre-training, and the set that the model has not observed. The paper demonstrates that detectors can more easily detect the OOD classes that the model was exposed to during upstream pre-training, compared to those that the model was not exposed to.

Critical strengths

The paper is well-written, with experiments carefully designed. Also, the paper involves several post-hoc OOD detectors, several models, showing a deep analysis.

Critical weaknesses

The main issue I believe is with the research question this paper addresses.

  1. Generally, in OOD detection, methods assume access to only ID data, such that the model is pre-trained on ID data, and then, scores like MSP, KNN distance are used for OOD detection.
  2. The assumption about OOD is that it is unknown and cannot be predicted in advance. So, again assuming that it can be available for upstream pre-training, or you know what class would be OOD, is not a valid assumption.
  3. Moreover, to evaluate the post-hoc OOD scores, the classifier is the same across all methods. Meaning that the effect of pre-training is already excluded. In more recent CLIP-based OOD detection, also, all methods are based on the same model CLIP. So, the question of how upstream pre-training can affect the performance is not relevant.
  4. The observation also is trivial and completely expected. It is very easy to see that pre-training on OOD data, can help to better detect that classes of OOD.
Questions for authors

Refer to weaknesses.

Reviewer_QQNmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the impact of outlier exposure in the upstream training, namely, whether outlier exposure impact the OOD performance. In the supervised setting, outlier exposure in training improves the OOD performance consistently across datasets, and also improves over most OOD detection methods. However, in the unsupervised setting, the results are mixed. The study is performed on three datasets, and using 9 OOD methods.

Critical strengths
  1. The impact of outlier exposure in upstream training is an interesting problem to study, because it affects the fairness of comparing OOD methods. This paper gives a quantitative evaluation on the impact.
Critical weaknesses

The setting of the upstream training is limited: (a) In the supervised setting, the OOD class is trained in the same way as an ID class. Whether we can call this setting an OOD detection is debatable, because OOD detection by definition, detects OOD samples. When they are included in training set, they are ID samples, and identifying them is a simple classification problem. It feels odd to drop the logit for that "OOD" class prediction probability and apply OOD methods for their detection because generally speaking, supervised classification are much more precise. (b) In the unsupervised setting, it is limited because backbone of the network is frozen after SSL, and only the last fc is tuned. It will be more practical to finetune the whole network after SSL. (c) Only one SSL method is evaluated, and the result is mixed without clear conclusion. It will be more comprehensive to include results for other SSL methods, such MAE.

Questions for authors
  1. In Tab. 1, GradOrth is clearly an outlier. Any insight for that?
  2. I think a more practical setting for both supervised and unsupervised OOD exposure is the following: Step 1. OOD exposure in pretrain, either supervised or self-supervised. Step 2. Fine-tune the whole model on ID classes only. Step 3. OOD detection on the fine-tuned model.
Reviewer_WN3VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how upstream supervised exposure impacts downstream OOD‑detection performance. Using a novel grouped rotating leave‑one‑out protocol, the authors fix downstream tasks, test data and model setups, and only toggle whether a future‑OOD class is supervised or withheld in upstream training. The work identifies OOD classes’ upstream training history as an under‑appreciated confounder for OOD benchmarks and offers practical evaluation guidelines. Limitations: labels vs data presence are not fully disentangled, geometry evidence is correlational, and tests are restricted to image classification. This is a well‑controlled study highlighting key methodological caveats for OOD‑detection research.

Critical strengths
  1. Carefully controlled experimental design.
  2. Comprehensive empirical coverage.
  3. Thorough reproducibility support.
Critical weaknesses
  1. The application scope seems narrow. upstream pretrain is classification CE loss, recent models such as CLIP and VLM, is contrastive loss or next token prediction loss. As to pure vision field, mask token prediction is a good upstream.
  2. For OOD detection field, we usually train classification-based OOD model from scratch, where authors also do not compare with methods that train-based OOD model from a classification-style pretrained.
  3. Only evaluates post‑hoc OOD detectors: The study focuses exclusively on post‑hoc OOD scoring on frozen encoders. It does not explore whether the provenance effect persists for methods that jointly train for OOD detection alongside classification. ,
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.