From Density to Detection: Diagnosing Likelihood Failures and Recovering OOD Signals

Likelihood-based out-of-distribution (OOD) detection assumes that high model density provides evidence of domain membership. Simple OOD images challenge this assumption across flow and autoregressive models. We examine the resulting failures at the level of individual-image decisions. Within-ID rankings reveal strong likelihood–compressibility alignment in Glow, PixelCNN++, and iGPT. Equal-entropy controls separate whole-image distribution entropy from visual complexity; training from scratch on complexity-filtered images then shows that excluded images outrank allowed images in $97.17\%$ of Glow comparisons. Noise-only training further reveals structured high-score preferences under a known population density. These controls distinguish limitations of an oracle-density rule from errors in the density learned by a model. We develop the consequences for detection through an $L$–$J$–$F$ decomposition: representation density, local volume change, and conditional variation along representation fibers. This framework extends the flow decomposition to general noninvertible networks and motivates learning from their component responses. Across 40 classifiers, the resulting LJF detector improves AUROC over its LF ablation by $2.01$, $5.00$, and $1.15$ percentage points on CIFAR-10, CIFAR-100, and ImageNet. Adding geometry also improves a strong 27-score fusion in 212 of 229 model–task comparisons under source-matched calibration. The findings clarify why raw density can mislead OOD decisions and show how its underlying model responses retain useful detection information.