- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the effect of upstream pre-training on OOD detection. The authors design an experiment, where the model is pre-trained on the given ID data and several sets of OOD classes, while a separate set of OOD classes is preserved and not used in upstream pre-training. Then, they evaluate the model using the ID data and both OOD sets, those the model has observed in upstream pre-training, and the set that the model has not observed. The paper demonstrates that detectors can more easily detect the OOD classes that the model was exposed to during upstream pre-training, compared to those that the model was not exposed to.
Critical strengths
The paper is well-written, with experiments carefully designed. Also, the paper involves several post-hoc OOD detectors, several models, showing a deep analysis.
Critical weaknesses
The main issue I believe is with the research question this paper addresses.
- Generally, in OOD detection, methods assume access to only ID data, such that the model is pre-trained on ID data, and then, scores like MSP, KNN distance are used for OOD detection.
- The assumption about OOD is that it is unknown and cannot be predicted in advance. So, again assuming that it can be available for upstream pre-training, or you know what class would be OOD, is not a valid assumption.
- Moreover, to evaluate the post-hoc OOD scores, the classifier is the same across all methods. Meaning that the effect of pre-training is already excluded. In more recent CLIP-based OOD detection, also, all methods are based on the same model CLIP. So, the question of how upstream pre-training can affect the performance is not relevant.
- The observation also is trivial and completely expected. It is very easy to see that pre-training on OOD data, can help to better detect that classes of OOD.
Questions for authors
Refer to weaknesses.