acceptodds
Withdrawn from ICLR 2027

Exploring a Principled Framework for Multi-label Out-of-distribution Detection with Pre-trained Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Out-of-distribution (OOD) detection enhances the reliability of machine learning models by identifying unexpected inputs from unknown classes. Recent progresses in large-scale vision–language pre-training have enabled zero-shot OOD detection using only in-distribution (ID) class names. Nevertheless, previous studies predominantly focus on detecting OOD inputs in the multi-class classification task, leaving the more realistic multi-label setting greatly under-explored. To mitigate this research gap, this paper investigates zero-shot multi-label OOD detection with pre-trained vision-language models. We first theoretically demonstrate that existing methods are fundamentally mismatched with the multi-label setting, as their scoring mechanisms fail to effectively aggregate evidence from multiple co-occurring ID labels. Motivated by this, we propose a probabilistic framework that formulates multi-label OOD detection through pairwise image-label association modeling, directly yielding a principled OOD scoring function that estimates the probability of an input being associated with at least one ID label in a training-free manner. Comprehensive evaluations on standard multi-label benchmarks, including MS-COCO and PASCAL VOC, empirically manifest that our method achieves state-of-the-art performance across a variety of OOD detection setups.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multi-label Out-of-distribution Detection, Vision-language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nzZzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses zero-shot multi-label OOD detection with pre-trained vision–language models. It argues that multi-class scores are mismatched to multi-label data and proposes a probabilistic formulation in which an input is OOD if it is associated with none of the ID labels. Under a noise-contrastive estimation framework, the detection score becomes a product of per-label posteriors, for which a Monte Carlo estimator and a consistency bound are given. On COCO and VOC, the method reports average FPR95 of 17.46% and 17.51%, improving over RenyiNeg by 6.82 and 9.03 points.

Critical strengths
  • The multi-label setting is genuinely under-explored, and the formulation in terms of image–label associations is well suited to its non-exclusive structure. It provides a clearer statement of the detection target than max- or sum-based aggregation over class scores.
  • The NCE derivation is technically clean. The label prior cancels, the posterior reduces to a tractable per-label expression, and the Monte Carlo approximation is stated explicitly, which makes the estimator easy to reproduce.
  • The empirical evaluation is broad and internally consistent. Improvements over NegLabel and RenyiNeg are observed across two ID datasets, four OOD datasets, three backbones, and two input resolutions, and the VOC FPR95 reduction from 26.54% to 17.51% is substantial.
Critical weaknesses
  • The score in Eq. 7 is not, in general, a probability. The identity Pr(∃y: C_y=1) = 1 − ∏_y q(C_y=0 | x) requires the association events to be conditionally independent given x, which is violated by co-occurring labels and by the shared image evidence on OOD inputs. The assumption is not stated, so the quantity should be presented as a noisy-OR surrogate rather than a calibrated probability.
  • The theoretical results do not establish the claimed fundamental mismatch. The bound in Theorem 1 is informative only when ε is small relative to τ log T; with τ=0.004 and typical cosine gaps of 0.01–0.05, the bound exceeds one. Theorem 2 describes generic softmax saturation and in fact shows that NegLabel benefits from secondary labels, which supports a conclusion of diminishing returns rather than structural incompatibility.
  • The experiments do not connect the theory to the observed behavior. The natural test—FPR95 and AUROC stratified by the number of ground-truth labels per image—is absent, as is an ablation of the aggregation function. The state-of-the-art claim is also selective: on COCO, JointEnergy (97.26/10.91) and MaxLogit (96.44/14.16) outperform the proposed method (96.39/17.46), so the claim should be restricted to training-free zero-shot methods.
Questions for authors

NA

Reviewer_kJG7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates multi-label out-of-distribution (OOD) detection using pretrained vision–language models (VLMs). The authors argue that existing methods are mainly designed for multi-class settings and cannot fully utilize information from multiple ID labels in an image. They analyze the label competition problem in MCM and the insufficient response of NegLabel to secondary labels. To address these issues, the authors construct an OOD detection score by aggregating association information between an image and multiple ID labels. The proposed method combines noise-contrastive estimation (NCE), energy-based modeling, and approximate computation using external labels to achieve training-free detection. Experiments on MS-COCO and PASCAL VOC, across different CLIP models and experimental settings, show that the method generally outperforms the main training-free baselines.

Critical strengths

S1. The research problem addressed in this paper is practically meaningful. The paper focuses on the coexistence of multiple ID labels in multi-label OOD detection and proposes a new scoring approach to address the difficulty of effectively aggregating information from multiple labels in existing methods.

S2. The experimental results are strong. The method clearly outperforms the main training-free baselines on MS-COCO and PASCAL VOC and achieves relatively consistent improvements across different OOD datasets, CLIP models, and input resolutions.

Critical weaknesses

W1. The core novelty of the paper is not sufficiently clear. The proposed product aggregation is similar in form to the existing noisy-OR aggregation, while NCE and energy-based modeling are also established techniques. Therefore, it remains unclear what specific new designs or advantages the proposed method offers over these existing techniques.

W2. Although the paper achieves strong experimental results, the experimental validation of its core mechanisms remains insufficient. Direct comparisons between different label aggregation methods and external label selection strategies are missing. The effects of the number of labels and the responses to secondary labels on detection performance are also not sufficiently analyzed. Therefore, it is still difficult to determine which design choices mainly contribute to the performance improvements.

Questions for authors

Q1. Could the authors further clarify the specific innovations of the proposed product aggregation compared with existing aggregation methods?

Q2. Could the authors provide additional ablation studies on the key design choices to clarify the contribution of each component to the performance improvements?

Reviewer_oqSWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies multi-label OOD detection. The paper first motivates the shortcomings of two important previous methods for multi-class OOD detection, to show that these methods inherently cannot support the multi-label case. Then, they propose their own multi-label score. Through experiments on two multi-label datasets they show the validity of their work.

Critical strengths
  1. The paper addresses an important and under-explored problem of multi-label OOD detection.
  2. The proposed method is interesting and novel.
  3. The experiments show improvement over previous studies, supporting the validity of the work.
Critical weaknesses

The main issue with this submission, I think, is that the paper is very hard to follow, and I believe the limitations mentioned regarding previous multi-class methods are, to some extent, overstated.

About the limitations described in Section 3 Motivation:

  1. From my understanding, Remark 2 states that when one of the ground-truth ID labels dominates, the contributions of the other ground-truth ID labels become saturated. However, I don't think this is a problem. As also stated in the paper, it is enough to consider the image as ID if the image matches only one ID label. So, I don't think the argument that this is a fundamental problem is justified, and some parts of the claim need to be relaxed, or additional conditions should be provided. I think even their own proposed score also suffer from this same problem, as their score in eq 19 also involves exponential function and normalization (similar to softmax), meaning that if one score dominates, the contributions of the others would become negligible.

  2. Theorem 1/Remark 1 for MCM considers cases where, for example, two ID labels match an image equally, while Theorem 2/Remark 2 considers cases where one label dominates. These are two different cases, and would be better to clearly describe them from the beginning. It is also important to describe how prevalent these two cases are or provide some image examples.

Clarity of the paper:

  1. The methodology section is very hard to follow, and there are too many details. Some terms are misleading. For example, the authors use the term negative pairs. However, negative labels are very common in the OOD detection literature, and the paper also describes this paradigm in the preliminary section. Therefore, using the term negative pairs without clearly distinguishing it from negative labels can be misleading.

  2. It would be good to provide an intuitive description of the final score, especially in comparison with the score used by NegLabel. How are the WordNet words used here different from NegLabel, and how is the score different from NegLabel?

Experiments:

  1. Comparison with "CMOOD: Concept-based Multi-label OOD Detection" is missing. This method is closely related, as it also addresses zero-shot multi-label OOD detection. It would be good to include it in the comparisons.

  2. Several recent multi-class studies are also missing, such as CSP (Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models), NegRefine (NegRefine: Refining Negative Label-Based Zero-Shot OOD Detection), and LAPT (LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models). It would be good to include these studies, too.

  3. While the paper includes JointEnergy, YolOOD is not included. I understand that YolOOD is not a direct baseline, as it considers the supervised setting. However, reporting JointEnergy without reporting YolOOD is not a good practice.

My main concerns are about the motivation section and the clarity of the proposed method (1 to 4). My current recommendation is Weak Reject, but I would be willing to increase it to Weak Accept if these concerns are adequately addressed.

Questions for authors

Please refer to Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.