- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes proper fusion of logit and feature contribution to final OOD detection score which works across different pretrained models. As logits and features live in different space, paper proposes scaling them with different scales and then combine them. If logit and feature information are not pointing towards same prediction then, the sample is more likely to be OOD. The paper also proposes to supplement the condition where the no of images available are less and then, it borrows the classifier information to get the class mean representation blended with images' mean representation.
Critical strengths
The paper provides the extensive experimental results. And the paper proposes to use the logit and feature agreement in the final OOD detection score. This might be a novel contribution of the paper. It also provides a way of estimating class center in limited ID data regime by leveraging the classifier weights. This is also useful thing when data is limited.
Critical weaknesses
The gains are near-trivial and the idea of the paper doesn't add much performance. The novelty is highly incremental and is additive of well known observations. Model sensitivity in OOD detection has been well documented. Performance variability across models is well documented and has also been attributed to feature norm. Though the evaluation is broader, there’s is not significant novel insights in the paper. ComboOOD also outperforms ONE in FPR metric in Table 7. The main text doesn’t discuss FPR metric and just centers the discussion around AUROC metric only as it doesn’t outperform ComboOOD in FPR metric. In table 3, again, comboood outperforms one in far column substantially.
Questions for authors
None.
3 comments
The gains are very trivial and the studies of eg. one image per class isn't a very realistic scenario. I thank authors for the response but I keep my original score.
Thank you for the feedback. We respectfully disagree on both points, for the reasons below.
Regarding the low-budget settings, one labeled image per class is an established evaluation setting: few-shot OOD detection methods are evaluated on ImageNet with one to sixteen labeled images per class, e.g., the prompt-learning method LoCoOp (Miyai et al., NeurIPS 2023). For a frozen classifier, such budgets arise when its training data cannot be used for calibration, for example, for a released model trained on proprietary data. Against the four baselines with the highest mean AUROC (HEAT, COMBOOD, RMDS++, and MDS++), ONE has the highest mean AUROC at every budget on 9 benchmark settings, and with two images per class it stays within 1.6 points of its full-calibration value, whereas these baselines lose 8 to 14 points (Table 6):
| Images per class | 1 | 2 | 5 | 20 | Full |
|---|---|---|---|---|---|
| HEAT | 79.2 | 76.0 | 84.5 | 85.9 | 86.3 |
| COMBOOD | 77.7 | 77.7 | 78.0 | 77.8 | 86.2 |
| RMDS++ | -- | 71.7 | 79.7 | 85.2 | 85.4 |
| MDS++ | -- | 77.6 | 83.3 | 84.7 | 85.5 |
| ONE | 84.6 | 85.3 | 86.0 | 86.6 | 86.9 |
--: undefined at one image per class.
Regarding the gains, the table below compares ONE with these four baselines, which have the highest mean AUROC, with full calibration (Table 2):
| 9 benchmark settings | 30 ImageNet models | |||||
|---|---|---|---|---|---|---|
| Mean AUROC ↑ | Mean regret ↓ | Max regret ↓ | Mean AUROC ↑ | Mean regret ↓ | Max regret ↓ | |
| HEAT | 86.3 | 2.4 | 4.6 | 83.7 | 3.5 | 10.1 |
| COMBOOD | 86.2 | 2.5 | 10.0 | 83.4 | 3.7 | 10.6 |
| RMDS++ | 85.4 | 3.4 | 9.3 | 83.3 | 3.8 | 10.9 |
| MDS++ | 85.5 | 3.3 | 7.5 | 83.1 | 4.0 | 16.0 |
| ONE | 86.9 | 1.8 | 4.4 | 85.6 | 1.6 | 6.7 |
Regret: AUROC gap to the best of the 25 scores on each setting or checkpoint (Eq. 11).
ONE has the smallest mean and maximum regret in both groups. On the 30-checkpoint panel, its mean regret is less than half that of each baseline, and each baseline falls 10 or more points behind the best score on some checkpoint, whereas ONE stays within 6.7.
We would be glad to address any further questions or concerns. Thank you.
We thank the reviewer for the constructive feedback. We address the novelty, the size of the gains, and FPR95 in turn; the revised PDF marks changes in red.
1. Novelty. "The novelty is highly incremental … Model sensitivity in OOD detection has been well documented." We agree that model dependence is well documented (Section 2); prior work mainly reports it or recovers performance by fine-tuning or retraining the network. Among post-hoc approaches, Mahalanobis++ traced the variation of Mahalanobis scores to the feature norm and removed it by normalizing the features; ONE adopts this normalization for its feature evidence. Beyond this normalization, ONE combines the classifier and the feature evidence within each class, checking that both favor the same class, and estimates the class references from classifier-derived priors. Both components have a measurable effect:
2. Gains. "The gains are near-trivial and the idea of the paper doesn't add much performance."
3. FPR95 and COMBOOD. "ComboOOD also outperforms ONE in FPR metric in Table 7 … In table 3, comboood outperforms one in far column substantially." FPR95 was in the appendix only for space; the revision places it next to AUROC (Table 3). On the 30-checkpoint panel, ONE has the lowest mean FPR95 and the smallest mean and maximum FPR95 regret; over all evaluated classifiers, including the panel, it has the lowest mean FPR95 (53.2; COMBOOD 55.2) and the lowest mean FPR95 regret (6.2; COMBOOD 8.2). COMBOOD's advantage in the far column comes from CIFAR far-OOD (AUROC 92.3 vs. 86.8), where it also uses intermediate-layer activations; on ImageNet, ONE is ahead on both near-OOD (84.1 vs. 81.9) and far-OOD (89.6 vs. 86.4; Table 4).