- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces OODConceptBench to evaluate open-world concept recognition in image–text dual encoders such as CLIP and SigLIP. Images are retrieved or generated for 560 concepts across seven categories. CLIP filters these images into L1, with low target-concept similarity pairs selected as L2. The paper evaluates 37 models or variants, measuring the fraction of image–concept pairs whose similarity exceeds a model-specific background threshold. All evaluated models score lower on L2. Later models are not consistently better, and overall improvements do not extend to every category.
Critical strengths
The paper organizes seven concept categories and compares 37 models under a common protocol. Pairs selected for low similarity by one CLIP model also prove difficult for other models, which provides some useful empirical evidence. However, it is really hard to read and follow.
Critical weaknesses
In my view, a benchmark paper should be a trustworthy ruler that yields useful new insights. This paper does not adequately establish either: the reliability of its image–concept labels is unverified, and the findings offer limited new understanding of model behavior.
-
The labels are not adequately validated. Section 4.1 retains concept assignments and retrieval noise, while subsequent CLIP filters do not check whether images contain the assigned concepts. No independent pairing accuracy or results on verified pairs are reported. Low scores may reflect wrong labels rather than recognition failures; testing 37 models does not resolve this.
- Search and generation tools, prompts, and label-assignment rules are insufficiently documented.
- Background calibration controls threshold exceedance on background images; it does not establish that exceeding the threshold measures concept recognition.
- Repeated findings and caveats add length, while the claim that filtering handles retrieval noise remains unsupported.
-
The L2 cutoff lacks justification, and the resulting findings offer limited insight. Appendix A.3 sets it to the maximum MNIST matching score plus one standard deviation (0.3031). Why MNIST, and why this formula? Sampling stability does not establish a meaningful difficulty boundary.
- CLIP-selected difficulty transfers across dual encoders, but this does not establish their concept blind spots or extend the findings to generative VLMs.
- What do “newer is not always better” and uneven category gains tell us beyond the reported scores?
- Figure 4 splits unsorted models across three columns, making comparisons needlessly difficult.
Questions for authors
- Were the final L1/L2 image–concept assignments independently checked? If so, please provide the audit procedure, accuracy by level and category, and any existing results on verified pairs.
- What theoretical or empirical evidence supports using the maximum MNIST matching score plus one standard deviation to define L2?
- Which search and image-generation tools were used? Are the queries, generation prompts, and label-assignment rules available?