- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TIE adds a "garbage" class to an image classifier. During training, a generator is trained against the classifier; the paper calls this "network inversion". The generated samples that the classifier is unsure about are added to the garbage class. At test time, the model rejects anything put in the garbage class, and the rest are checked using a Mahalanobis score. All models are small CNNs trained from scratch on MNIST, FashionMNIST, SVHN, CIFAR-10/100 and TinyImageNet. The paper claims a near-0% FPR on MNIST and FashionMNIST. For CIFAR-10, it reports an average FPR of 7.59, compared to 7.46 for Dream-OOD.
Critical strengths
A simple recipe: it needs no outlier data and no pretrained generator, and it adds no test-time cost.
Critical weaknesses
First and foremost, the experiments are very lacking. It feels like the paper is from ten years ago, when we were still working on classifiers on MNIST and CIFAR-10. Even introduction to computer vision classes don't use MNIST and CIFAR-10 right now and start with ImageNet.The paper basically ignores how OOD detection is done today. Modern top detectors use pretrained models. Zero-shot detectors built on vision-language models (VLMs) do not even need training on the original data:
- CIFAR-10. Zero-shot CLIP detectors get 94.8 near-OOD and 99.3 far-OOD AUROC (AdaNeg, NeurIPS 2024, OpenOOD benchmark). This is better than a ResNet-18 trained on CIFAR-10, which gets 88.0 / 90.7 with MSP (OpenOOD v1.5).
- CIFAR-100 vs. CIFAR-10. Pretrained ViTs raise AUROC from 85% to over 96% (Fort et al., NeurIPS 2021).
- ImageNet-1K. The zero-shot NegLabel reaches 94.21 AUROC (ICLR 2024).
TIE instead trains small CNNs from scratch for 20 epochs. They reach only 85.5% accuracy on CIFAR-10, and the images are never larger than 64×64. TIE does not compare itself with any of these modern methods. Its "≈0 FPR" claim comes from MNIST and FashionMNIST, a setting that is close to saturated: with MNIST as the training data, plain MSP already reaches 98.5 far-OOD AUROC in OpenOOD v1. TIE also requires training the classifier and a generator from scratch, so it is unclear whether it can work with pretrained models.
W2. Limited novelty:
- The extra OOD class. Handling unknown inputs with "an additional background or garbage class" is an old idea, and Dhamija et al. (NeurIPS 2018) found it "generally insufficient when previously unseen classes are encountered". Further examples:
- Lee et al. (ICLR 2018, "Training confidence-calibrated classifiers for detecting out-of-distribution samples", App. E) tested an explicit (K+1)-th OOD class and found it often significantly worse than their confidence loss.
- A classifier with an extra noise class is a standard baseline (Ren et al., NeurIPS 2019).
- Vernekar et al. (NeurIPS 2019 workshop) already train an (n+1)-class classifier on generated OOD samples, on MNIST and FashionMNIST.
- Filling that class with generated samples. Training the extra class on samples from a generator that targets the classifier is the (K+1)-class "fake" discriminator of semi-supervised GANs (Salimans et al., NIPS 2016). It is also Nguyen et al.'s fooling-image retraining (CVPR 2015).
- The new part. Uncertainty-guided exclusion is never tested on its own (no ablation study). It is also missing from the provided notebook, which adds every inverted sample to the garbage class and sets the KL and cosine loss weights to zero.
None of these prior works is cited.
Questions for authors
- How does TIE compare with zero-shot VLM detectors (such as MCM or NegLabel) and with pretrained models on the OpenOOD CIFAR-10 and ImageNet-1K benchmarks? Can TIE be used on top of a pretrained model?
- What percentage of normal test images does the garbage class wrongly reject?