- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes OptLabel, a framework for ID-supervised OOD detection based on negative labels in CLIP's embedding space. The method consists of two stages: first, it optimizes positive label embeddings using cross-entropy on labeled ID data; second, it freezes the optimized positive labels and optimizes negative label embeddings using both ID features and synthetic OOD features. The synthetic OOD features are generated by applying isotropic Gaussian perturbations to normalized ID embeddings, followed by renormalization. The paper provides a geometric analysis of the resulting angular displacement and argues that this simple synthesis strategy can generate useful near-boundary OOD supervision without additional filtering. Experiments verify the advantages of the proposed method.
Critical strengths
- The paper addresses a relevant problem in negative-label-based OOD detection and reports consistent improvements over existing methods.
- The method is simple and efficient, requiring neither CLIP fine-tuning nor external OOD data or image generation models.
Critical weaknesses
- The methodological novelty is limited, as the main components, including Gaussian feature perturbation, label embedding optimization, and two-stage training, are largely based on established techniques. The method primarily combines these existing ideas, and the geometric analysis does not sufficiently establish a substantial methodological advance over prior work.
- The negative labels are optimized against Gaussian-perturbed ID features, whose distribution may differ substantially from that of real OOD samples. The paper provides limited evidence explaining why the learned separation should transfer to unseen OOD distributions.
- Perturbing every ID feature without considering local density or class structure may generate samples that still belong to the ID distribution. Treating all such samples as OOD could introduce conflicting supervision, especially in regions where ID classes overlap.
- Directly optimizing negative-label embeddings in CLIP's feature space may weaken their original semantic meaning. The paper does not examine whether the optimized negative labels retain meaningful semantic information or simply become learned discriminative prototypes.
Questions for authors
- What is the key methodological contribution of OptLabel beyond combining existing techniques? In particular, how does the proposed approach fundamentally differ from prior feature-space OOD synthesis and label optimization methods?
- What evidence supports the assumption that a detector trained with Gaussian-perturbed ID features can generalize to real, unseen OOD distributions, given the potential distribution gap between synthetic and real OOD samples?
- How do the authors ensure that perturbing every ID feature does not generate samples that still belong to the ID distribution? Could such samples introduce conflicting supervision, particularly in overlapping or high-density ID regions?
- Do the optimized negative-label embeddings retain their original semantic meaning, or do they effectively become learned discriminative prototypes? Can the authors provide evidence that the semantic information remains important after optimization?
2 comments
Response to Question 3
At first glance, it seems that perturbing every ID image may generate synthesized OOD samples that move closer to other ID samples, potentially introducing conflicting supervision. To address this concern, NPOS applies a filtering step based on k-NN distances to ID samples to remove problematic synthesized samples close to ID data. One of our contributions is the theoretical analysis that shows that this additional filtering is unnecessary in a high dimensional space like CLIP's.
Theoretically, we showed that for a synthesized OOD sample from perturbing an ID anchor , and for any other ID sample , the approximation (see Equation 6) holds in high dimension space with high probability, where the error term diminishes with high dimension . See Equation 11 and Appendix A.2 for the derivation of Equation 6. Essentially, this result says that the similarity between the synthesized sample and any other ID sample is reduced by the factor (which is between 0 and 1), compared to the similarity between the two ID samples and , thus, the synthesized sample is not closer to than the two ID samples and . This theoretically shows that in the case of high dimension , a synthesized sample moves away from all of ID samples, instead of moving away from one while getting closer to another. A sufficiently large angular displacement is needed to move synthesized samples beyond the ID region. We selected the angle empirically as shown in Figure 4 and works well for all tested datasets and settings.
The intuition for a synthesized sample not only moving away from its ID anchor but also from the other ID sample is that the Gaussian perturbation places the synthesized sample randomly on an angular shell at the angle from its ID anchor (see the red circle in Figure 2). In a high dimensional space, there are many possible directions on this shell, making a random perturbation unlikely select a direction that would bring the synthesized OOD sample closer to another ID sample , because random directions in high-dimensional spaces tend to be nearly orthogonal. In contrast, this property does not hold in low-dimensional spaces. For example, in 2D space, there are only two possible directions along the unit circle, and a small Gaussian perturbation has a large chance of bringing the synthesized sample closer to another ID sample.
The above result is also supported empirically on the ImageNet-1K benchmark. Figure 4 shows that keeping all synthesized OOD samples without filtering (100% kept) achieves the lowest (best) average FPR95, and filtering synthesized samples (based on their -NN distances to ID data as performed in NPOS), consistently leads to larger (worse) average FPR95 across all tested values of and all filtering ratios (keeping 50% to 90% of synthesized OOD samples). The same trend holds when using synthesized samples (Appendix Figure 7).
In summary, our theoretical analysis makes three contributions: (i) it simplifies significantly the OOD synthesis process in high dimension case by removing the need for considering local density or filtering synthesized samples, (ii) it improves OOD detection by keeping all synthesized samples (i.e., data for learning), and (iii) it contributes new knowledge to the OOD synthesis literature.
Response to Question 4
Retaining the semantic meaning of individual negative labels is not required by the OOD detection problem. As defined by the OOD detection problem, it is sufficient to detect whether a sample is OOD; it is not required to tell exactly what negative label corresponds to the OOD sample. In other words, our negative labels are used as learned discriminative prototypes as the reviewer called it.
We sincerely thank the reviewer for taking the time to review our work and for providing insightful comments. We have made our best effort to address all the comments and would greatly appreciate any further questions or feedback from the reviewer.
Response to Question 1
Our methodology goes beyond primarily combining existing techniques/ideas. Each component differs from existing techniques in several major ways:
On 2-stage training: A straightforward way of applying label optimization, which is close to the idea of classifier prototype learning, to our negative-label based setting would be the joint-optimization of positive and negative labels in a single loss. However, this joint-optimization is outperformed by our 2-stage training that first optimizes positive labels then optimizes negative labels while freezing optimized positive labels, effectively preventing distorting the optimized positive-label representations (see Table 4, lines 462-465). Prior to this study, it is not clear which order of optimization works best.
On label optimization: SynOOD trains an additional projection layer over CLIP's image encoder and optimizes only a subset of negative labels. In comparison, by optimizing the label embeddings generated by the text encoder without introducing an additional layer, we preserve CLIP's original representations and generalization capabilities. Prior prompt-tuning methods optimize the context tokens of a class label in CLIP's input space while keeping class label itself fixed, for example, in the prompt "a photo of a [dog]", the context tokens "a photo of a" are replaced with learnable vectors. This context token is shared for all classes but the actual class (e.g., [dog]) is not optimized. We directly optimize individual class label's embeddings in CLIP's output space, examining a larger optimization space and allowing the thousands of negative labels to be updated efficiently.
On OOD synthesis: Our analysis theoretically guarantees that in high dimensional space (like CLIP's) the synthesized OOD sample from every ID image moves away from all ID samples with high probability, therefore, there is no need to consider local density or additional filtering. A more detailed explanation is given in our response to your Question 3. This property of Gaussian perturbation in high dimensional space was previously unknown in the OOD synthesis literature, for example, NPOS uses a filtering step to remove certain synthesized samples. Our method differs from recent OOD synthesis methods, like SynOOD, and few-shot methods LoCoOp, and Local-Prompt, that derive OOD supervision from non-ID concepts or crops in ID images, limiting their OOD coverage to the background content present in the ID train images (lines 90–93, 147–149), as demonstrated empirically on the ImageNet subsets (Table 2, lines 399–404). Unlike these methods, we synthesize OOD samples directly in the embedding space, which generalizes better to the unseen OOD distribution, as explained in the next response.
Response to Question 2
The generalization to real, unseen OOD distributions is supported along the following lines:
First, several prior studies, e.g., VOS, NPOS, SynOOD, have reported that OOD samples near the ID boundary are sufficient for learning ID/OOD separation, and OOD samples far from the boundary (i.e., in the interior of OOD regions) provide no additional useful information. Similar to these studies, our sampling is not to uniformly cover the entire real OOD region, but to synthesize challenging near-boundary OOD samples that help learn an effective separation.
Second, our negative labels (for representing OOD space) is initialized using the WordNet-based labels suggested by prior zero-shot methods, which represents a wide range of semantic OOD concepts and provides a broad coverage of the OOD space. Table 4 supports the importance of this semantic initialization: replacing it with random initialization increases average FPR95 from 21.72 to 28.51.
Third, empirical evidence for this generalization comes from our experiments across diverse ID/OOD settings, where the test OOD data are real and unseen to our label optimization and OOD synthesis, and these settings cover both near-boundary and far-boundary OOD datasets (Tables 1–2). Our method consistently achieves substantial improvements over prior methods in the same ID-supervised setting. See additional empirical evidences in our response to Weakness 3 of Reviewer PN3Z.