Scaling Pretrained Representations Enables Label-Free Out-of-Distribution Detection Without Fine-Tuning

Models trained with deep learning often fail to signal when inputs fall outside their training data manifold, leading to unreliable predictions under distribution shift. Prior work suggests that effective out-of-distribution (OOD) detection often requires some combination of class-conditional modeling, sophisticated detectors, and specialized models obtained through supervised fine-tuning. We ask whether, in modern pretrained models, label-free OOD detection is increasingly limited by the representation being probed rather than the sophistication of the detector. Across 59 backbone–task pairings spanning vision and language, we compare two complementary label-free detectors: a global Mahalanobis estimator fit on unlabeled latent representations, and ReSCOPED, a lightweight, diffusion-based typicality estimator operating on the same features at a local level. As pretrained representations scale, both detectors improve substantially, previously challenging benchmarks approach saturation even with simple label-free detection, and the performance gap between the two methods shrinks. This same trend persists on harder, non-saturated vision and language benchmarks. These results suggest that label-free OOD detection depends strongly on the geometry exposed by frozen pretrained backbones, reducing the importance of detector choice as backbone scale increases and enabling efficient deployment directly on frozen models.