- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper addresses likelihood-based OOD detection. Controlled diagnostics on generative models (Glow, PixelCNN++, iGPT) show that likelihood favors simple images, which the authors attribute to model misestimation rather than to entropy or typical-set effects. The paper then proposes an L–J–F decomposition of the log-density (representation density, Jacobian volume, fiber term), instantiated on frozen classifiers and combined by a logistic regression trained on labeled ID and OOD data. Experiments cover 40 classifiers on CIFAR-10, CIFAR-100 and ImageNet, compared with 27 existing OOD scores and their fusion, plus a flow-based counterpart on CelebA vs. SVHN.
Critical strengths
- Diagnising & Understanding likelihood failures in OOD detection is a relevant, long-standing question.
- Some diagnostics are carefully controlled. In particular, the excluded-support experiment cleanly separates model misestimation from typical-set explanations.
- The classifier evaluation is broad (40 models, six density estimators), with a fusion baseline of 27 score families.
Critical weaknesses
-
Presentation and clarity. The paper is difficult to follow, which is a serious issue for a top-tier venue.
- The method is never stated plainly in the main text. There is no contribution list and no overview figure. The actual detector, a logistic regression on three scores, only appears explicitly in the appendix (Eq. 30).
- The text is saturated with defensive qualifications (“this does not establish…”, “this is not a claim that…”), which bury the main message.
- Notation is overloaded: L and J denote different quantities for flows and for classifiers. Several terms are used without definition (“source-matched calibration”, “readout”, “component footprint”, “higher-score fraction”).
- Space is poorly allocated: elementary identities (Eqs. 2–3, the Gaussian example) take main-text space, while the estimators and the evaluation protocol are relegated to the appendix.
-
Unclear focus: generative diagnostics vs. a classifier detector. The paper combines two largely independent contributions.
- The diagnostics concern the simplicity bias of generative likelihoods.
- The proposed detector operates on classifiers and uses no generative likelihood, so the diagnosed failure plays no role in it.
- The coarea decomposition is the only bridge, and it is not operational for classifiers (next point).
- The flow-side detection results rest on a single setting (CelebA vs. SVHN).
-
The decomposition is not what is estimated. The identity log p_X = L + J + F holds for a specified input density p_X, which a classifier does not provide.
- L and F are Gaussian fits in feature space, whereas J is computed in input space, so the three terms do not refer to the same density.
- F is fitted unconditionally on a feature residual, while the exact F is conditional on the input fiber. The paper’s own Appendix F.1 (Eq. 20) shows that a feature-space conditional does not identify the input-space F.
- Consistently, the fixed sum of components does not work (CIFAR-100 AUROC 63.18 for J+L+F vs. 63.56 for L+F); only the learned weighting helps. The decomposition is therefore a motivation rather than the basis of the estimator.
-
Limited novelty.
- Most diagnostic findings confirm known explanations: the compressibility link (Serrà et al., 2020), the density-vs-typicality distinction (Nalisnick et al., 2019b; Le Lan & Dinh, 2021), and model misestimation (Zhang et al., 2021).
- On the flow side, the radial recalibration is borrowed (Dheur & Ben Taieb, 2025). The two-sided test is a conformal version of typicality testing. Component-based flow detectors already exist (DoSE, Morningstar et al., 2021; likelihood-free detection, Ahmadian & Lindsten, 2021).
- On the classifier side, L resembles a Mahalanobis score on logits and F a ViM/Residual score, leaving the Jacobian term J as the only new ingredient.
-
Reliance on labeled OOD data from the test source. The detector’s weights and signs are learned on labeled OOD images from the same source as the test OOD images.
- This turns OOD detection into binary classification against a known distribution, whereas the defining difficulty of OOD detection is that test-time shifts are unknown: OOD samples are by definition anything outside the ID distribution and cannot be collected representatively. In my view, this is a dead end by construction: a detector tuned to one OOD source offers no guarantee for the next one.
- Each OOD dataset is split into calibration and test halves, so the detector is trained and evaluated on i.i.d. samples of the same OOD dataset. At the detector level, there is no distribution shift at all.
- The fixed-sum results above show that the components only help once their weighting is learned on the target OOD, which is precisely the information unavailable at deployment.
- The strongest baseline (ALL) is fitted in the same regime, so the reported gains hold only in this unrealistic setting. An OOD-free variant of LJF, compared with standard unsupervised detectors, is needed. At a minimum, a cross-source evaluation (e.g., leave-one-source-out) would show how much of the gains depends on source matching.
-
Weak experimental results.
- Even in this favorable supervised setting, LJF alone remains below the best single baseline selected on calibration data: 88.87 vs. 90.65 AUROC on CIFAR-10, 82.30 vs. 82.94 on CIFAR-100, and 80.57 vs. 88.13 on ImageNet. The LJF-over-LF gains are thus gains over a weak ablation.
- The only improvement over existing methods is +0.39 AUROC when adding J to the 27-score fusion. It is concentrated on a few easy targets (e.g., SVHN for CIFAR-100) and negligible elsewhere (e.g., +0.01 on NINCO, +0.03 on OpenImage-O for ImageNet).
- On ImageNet, the LJF-over-LF interval for the primary estimator includes zero.
- The cost of computing the full input-to-logit Jacobian, which forces an arbitrary 32-dimensional logit projection on ImageNet, is not discussed.
Questions for authors
See weaknesses