acceptodds
Withdrawn from ICLR 2027

What Remains OOD? OODConceptBench for Open-World Concept Recognition in Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In the era of large-scale pretrained models, zero-shot generalization has increasingly blurred the boundary between known and unknown concepts. However, current evaluations do not directly reveal which open-world concepts different vision–language models can recognize. To study this problem, we introduce OODConceptBench, a hierarchical benchmark for open-world concept recognition on image–concept pairs. The full benchmark set (L1) contains 30,752 images, and the harder subset (L2) contains 14,285 images. The benchmark covers 560 concept phrases across seven open-world categories. We evaluate 40 vision–language models across release year, model family, and variant. We obtain three main findings. First, current models still show a performance gap for open-world concepts. Second, later-released models do not necessarily achieve stronger open-world concept recognition. Third, an increase in overall recognition does not imply consistent improvement across all benchmark categories. Together, these results show that OOD remains a challenge even in the era of large-scale vision–language pretraining.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, open-world concept recognition, out-of-distribution, OOD benchmark, zero-shot recognition, datasets and benchmarks, hierarchical benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7GanOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces OODConceptBench to evaluate open-world concept recognition in image–text dual encoders such as CLIP and SigLIP. Images are retrieved or generated for 560 concepts across seven categories. CLIP filters these images into L1, with low target-concept similarity pairs selected as L2. The paper evaluates 37 models or variants, measuring the fraction of image–concept pairs whose similarity exceeds a model-specific background threshold. All evaluated models score lower on L2. Later models are not consistently better, and overall improvements do not extend to every category.

Critical strengths

The paper organizes seven concept categories and compares 37 models under a common protocol. Pairs selected for low similarity by one CLIP model also prove difficult for other models, which provides some useful empirical evidence. However, it is really hard to read and follow.

Critical weaknesses

In my view, a benchmark paper should be a trustworthy ruler that yields useful new insights. This paper does not adequately establish either: the reliability of its image–concept labels is unverified, and the findings offer limited new understanding of model behavior.

  1. The labels are not adequately validated. Section 4.1 retains concept assignments and retrieval noise, while subsequent CLIP filters do not check whether images contain the assigned concepts. No independent pairing accuracy or results on verified pairs are reported. Low scores may reflect wrong labels rather than recognition failures; testing 37 models does not resolve this.

    • Search and generation tools, prompts, and label-assignment rules are insufficiently documented.
    • Background calibration controls threshold exceedance on background images; it does not establish that exceeding the threshold measures concept recognition.
    • Repeated findings and caveats add length, while the claim that filtering handles retrieval noise remains unsupported.
  2. The L2 cutoff lacks justification, and the resulting findings offer limited insight. Appendix A.3 sets it to the maximum MNIST matching score plus one standard deviation (0.3031). Why MNIST, and why this formula? Sampling stability does not establish a meaningful difficulty boundary.

    • CLIP-selected difficulty transfers across dual encoders, but this does not establish their concept blind spots or extend the findings to generative VLMs.
    • What do “newer is not always better” and uneven category gains tell us beyond the reported scores?
    • Figure 4 splits unsorted models across three columns, making comparisons needlessly difficult.
Questions for authors
  1. Were the final L1/L2 image–concept assignments independently checked? If so, please provide the audit procedure, accuracy by level and category, and any existing results on verified pairs.
  2. What theoretical or empirical evidence supports using the maximum MNIST matching score plus one standard deviation to define L2?
  3. Which search and image-generation tools were used? Are the queries, generation prompts, and label-assignment rules available?
Reviewer_1ASvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the recognition scope of large-scale pretrained VLMs is difficult to characterize with a fixed class set alone, and proposes OODConceptBench to evaluate open-world concept recognition. Recognition is judged by whether the similarity between an image and the text of its concept exceeds a model-specifically calibrated threshold, which defines a model-specific OOD boundary. The benchmark is built from 560 concept phrases in seven categories using web search and image generation. It consists of L1, 30,752 images that pass CLIP-based filtering, and L2, a subset of 14,285 L1 images with lower similarity to their target concepts. Across 37 VLMs and variants, recognition on L2 is lower than on L1 for every model, and later-released models do not necessarily achieve higher recognition. The paper also shows that gains in overall recognition do not imply improvement in every category, and argues for evaluating recognition at the level of individual models and concept categories.

Critical strengths
  1. A useful evaluation perspective beyond fixed label sets. Large-scale pretrained VLMs can recognize concepts outside predefined class taxonomies, making label-set membership insufficient to characterize their recognition scope. The paper evaluates recognition on shared image–concept pairs across models, providing a useful perspective for identifying which concepts remain challenging for different VLMs.

  2. Broad concept and model coverage enables informative analysis of recognition weaknesses. The benchmark is constructed from 560 concept phrases across seven categories and evaluates 37 models and variants under a common protocol. Its category-level analysis reveals potential weaknesses obscured by aggregate scores, particularly that improvements in overall recognition can coexist with declines in individual categories.

Critical weaknesses
  1. Insufficient detail on target-concept verification.
    • Section 4.1 reports a manual review for technical usability of the collected images and states that this review preserves the concept assignments made during data collection. The paper does not describe whether, or how, each retrieved or generated image was verified to depict its assigned concept, nor how ambiguous or partially matching examples were handled.
    • This is particularly important for L2, which deliberately retains pairs with low image–text similarity. Such pairs may be genuinely difficult examples of the target concept or incorrect assignments, and the similarity-based filters cannot distinguish the two. Without a description of semantic verification, or an audit of a sample of L2 pairs, it is difficult to determine whether the reported recognition failures reflect model limitations or annotation noise, which affects the interpretation of the subsequent experiments.
  2. Limited insight from the main findings.
    • Finding 1 largely confirms the intended effect of benchmark construction. L2 is explicitly selected for low target-concept similarity under CLIP ViT-B/32. Its lower recognition across other models is useful evidence that this difficulty transfers, but provides limited additional insight as a main finding. An analysis showing whether these failures reflect difficulty distinguishing the target concept from plausible alternatives would more directly validate the benchmark’s intended purpose.
    • The release-year analysis (Finding 2) provides limited insight as a main finding. Release year is a weak proxy for model capability, and the year groups are small and unbalanced (5/9/16/3/4 models) and differ in architecture and training data. Relating recognition rates to established capability measures, such as zero-shot classification accuracy or retrieval performance, would be more informative, as it would show whether OODConceptBench captures limitations beyond general model capability.
Questions for authors
  1. How was image–concept alignment verified during dataset construction? Section 4.1 mentions manual review; could you clarify whether this included checking that images depict their assigned concepts, what criteria were used, and how ambiguous or partially matching examples were handled?
  2. Why was a threshold-based recognition criterion adopted instead of a ranking-based one? A ranking-based evaluation, using the existing L1 concept vocabulary as a fixed candidate set, would directly test whether a model prefers the target concept over the alternatives. Could you report top-1 and top-5 accuracy alongside the threshold-based recognition rate, and check whether the main model comparisons and the L1–L2 gap remain consistent?
  3. Do the main results hold if the filtering model, OpenAI CLIP ViT-B/32, is changed? All three filtering rules use this single CLIP model, and L2 in particular consists of samples to which it assigns low similarity to the correct concept. The drop on L2 across 37 models is meaningful, but it remains to be confirmed whether it reflects a general difficulty of concept recognition or a CLIP-based selection bias shared across models. Why was CLIP ViT-B/32 chosen as the filtering model, and could you show whether the L1–L2 difficulty gap and the main model- and category-level comparisons are preserved when the same candidate pool is filtered with a different model, such as one from the SigLIP family?
Reviewer_pkfcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OODConceptBench, a benchmark for evaluating open-world concept recognition in vision-language models. It constructs two evaluation sets (L1 and L2) covering seven concept categories and evaluates 37 pretrained models using model-specific similarity thresholds. The main findings concern the transferability of difficult samples, non-monotonic performance across release-year groups, and inconsistent improvements across concept categories.

Critical strengths
  1. The paper investigates model-specific recognition boundaries beyond predefined ID/OOD categories, providing a useful perspective on open-world VLM evaluation.
  2. The benchmark covers diverse concepts and evaluates 37 models under a unified protocol. The consistently lower L2 recognition rates across models provide useful evidence of cross-model difficulty transfer.
Critical weaknesses
  1. The image collection process mainly includes technical usability checks, without sufficient independent verification of image–concept correctness. This is particularly concerning for L2, where low similarity may reflect incorrect concept assignments rather than genuine recognition difficulty.
  2. Although the experiments reveal differences in model-specific recognition boundaries, all filtering rules rely on CLIP ViT-B/32, and L2 is selected using low image–concept similarity. This raises concerns about construction-model bias and the independence of the observed difficulty.
  3. The 2024 group consists entirely of DFN-CLIP variants, while the 2025 group consists entirely of SigLIP2 variants. Thus, the reported temporal trends are confounded by model-family differences.
  4. The work primarily offers a reformulation of OOD evaluation using existing similarity measures and calibration techniques. Its claim of being the first systematic investigation of OOD concepts in VLMs is insufficiently justified.
Questions for authors
  1. Is there independent human validation of image–concept correctness, particularly for L2, to distinguish genuine recognition difficulty from potentially incorrect concept assignments?
  2. How would the composition of the L1 and L2 subsets change if a different model were used for benchmark construction instead of CLIP ViT-B/32?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.