acceptodds
Under review as a conference paper at ICLR 2027

Classify by Descriptions: A Reality Check

Abstract

Classify by Descriptions (CbD) has been shown to both improve zero-shot image classification and provide natural language explanations of predictions. Instead of relying on costly per-task human expert training data to achieve this, CbD infers per-class descriptors from large language models (LLMs) and then uses a vision-language model (VLM) to recognize their appearance in a query image. In this work, we critically analyze CbD and characterize its behavior when recognition becomes more fine-grained and target categories might be underrepresented in LLM and VLM pre-training data. Existing evaluation setups are unable to support this analysis with either limited coverage of realistic problem domains or a single granularity of classification. To address this gap, we introduce the classification benchmark TaxoBench, a benchmark spanning 7 realistic classification domains and multiple levels of classification granularity. Using TaxoBench, we find that CbD performance drops substantially as categories become more fine-grained and less familiar to the underlying VLM. We further identify two key sources of failure: LLM-generated descriptions often fail to capture the visual distinctions required by the task, and VLMs struggle to reliably recognize the resulting open-vocabulary concepts. We evaluate existing and new mitigation strategies but find only modest improvements, emphasizing the need for further research in more robust pre-training of foundational models and CbD as a whole.

Primary area:
interpretability and explainable AI
Keywords:
Concept Bottlenecks, Zero-shot Classification, Vision-Language Models, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Fzn3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors are motivated by the unreliability of current benchmarks for Classify by Descriptions (CbD). Current benchmarks feature limited domain coverage and limited coarse-to-fine recognition coverage. To address this problem, the paper introduces TaxoBench, providing broad coverage of realistic classification problems, common and uncommon category names, and both coarse- and fine-grained recognition problems. TaxoBench is used to pose and answer three important questions. Evaluation with TaxoBench shows that current approaches struggle with both producing good descriptions and recognizing these descriptions. Comparison between CbD and black-box methods confirms that zero-shot performance relies on the overlap between unseen classes and pre-training data, and that descriptions are not useful even in fine-grained scenarios. Further analysis identifies two major bottlenecks: description generation and recognition ability. Different prompting strategies and entry-level categorization improve descriptor quality, while existing optimization methods show limited gains. In terms of recognition ability, more powerful recent CLIP-based models provide modest improvements, while mLLMs achieve substantial gains. Overall, the results highlight the limitations of current CbD approaches and the need for better description generation and recognition.

Critical strengths
  1. The paper introduces a new benchmark, TaxoBench. The motivation behind TaxoBench and the research questions are clear, well-justified, and interesting. The paper clearly identifies three key limitations of CbD: 1) Reliance on class names, 2) Language bottleneck, and 3) Recognition bottleneck.

  2. TaxoBench covers diverse real-world domains and multiple levels of classification granularity, enabling a more comprehensive evaluation of CbD across different domains and difficulty levels.

  3. The paper presents a well-designed experimental setup and a thorough analysis of the results, providing valuable insights into the fundamental limitations of both description generation and recognition in CbD.

Critical weaknesses
  1. In Table 2, the effectiveness of CbD varies significantly across different models. While CLIP and OpenCLIP benefit from CbD, BioCLIP2 shows a substantial performance drop. Moreover, the contribution of descriptions appears limited compared to that of class names in most settings. These observations raise questions about whether the results sufficiently support the claimed recognition bottleneck. Can the recognition bottleneck be clearly disentangled from the low quality of generated descriptions? Although Section 6.1 compares different VLMs using the same descriptions, the recognition bottleneck is not sufficiently disentangled from the language bottleneck. Evaluating whether similar results are obtained with higher-quality descriptions would provide stronger evidence for the claimed recognition bottleneck.

  2. In line 241, the authors suggest that one reason CbD relies heavily on class names is their familiarity from pre-training. However, VLMs such as CLIP are pre-trained on large-scale web-crawled image-text pairs, which may contain not only class names but also rich natural language descriptions. Therefore, it is unclear whether the observed behavior can be attributed specifically to class name familiarity without further analysis of the pre-training data.

  3. The analysis of the two prompting strategies in Table 4 is limited. While full-sentence prompting largely improves performance on everyday item classification, the improvements from few-shot in-context prompting are more pronounced in species classification. However, according to the analysis in Section 4, species classification tasks particularly benefit from descriptions. Thus, despite full-sentence prompting providing more detailed descriptions, further analysis is needed to explain why it is less effective in species classification. A deeper analysis of these results would strengthen understanding of how description quality affects CbD performance.

  4. In Section 6.2, it is unclear whether mLLMs truly address the recognition bottleneck or simply benefit from their rich general world knowledge. The limited improvements in Section 6.3 further raise questions about whether the gains from mLLMs can be attributed primarily to better descriptor recognition.

Questions for authors
  1. Why does CbD with BioCLIP2 perform substantially worse than black-box zero-shot classification even when class names are included? While the performance degradation without class names is reasonably explained, the degradation with class names is not sufficiently investigated. Is this due to the language bottleneck or the recognition bottleneck?

  2. Do full-sentence descriptions actually contain entry-level category information?

4 comments
AuthorsOpenReview ↗

Review Response

We thank you for your insightful feedback and hope to address your comments and concerns below.

Weaknesses:

  1. Please see overall response #3 titled “Disentangling VLM recognition performance from LLM-generated descriptor quality.”

  2. Please see overall response #1 titled “Pre-training coverage analysis.”

  3. In our subsequent analysis, we found that full-sentence prompting mainly helps everyday item domains by reintroducing category names that CLIP knows well into descriptors. In fact, when we mask class names from descriptors and replace them with generic super categories, we find the overall performance on everyday item domains drops from 44.4% and 69.2% to 42.2% and 58.4% on fine and intermediate tasks, respectively. This is strictly less than the corresponding performance of masked descriptors from few-shot prompting (i.e., 42.9% and 59.5%). The species domain similarly introduces organism names. However, unlike the everyday item domains, these names are typically scientific abbreviations or Latin binomials (e.g., A. aequalis) which show up much less frequently in standard web-scraped internet text captions. In fact, when we masked out names from few-shot and full-sentence descriptors, performance dropped no more than 1 percentage point across class, order, and family levels. On the other hand, few-shot prompting helps ground descriptors, especially for long-tailed class names. For example, the sea anemone “Heteranthus verruculatus” initially was mistaken for a plant with descriptors like “The plant has a shrubby or small bush-like form with multiple stems.” and “Leaves are typically opposite, simple, and may have a slightly rough or textured surface.” When provided species observations in context, the produced descriptors properly identify sea anemone visual features like “Tentacles often have a warty or knobby appearance.” and “Tentacles arranged radially around the central mouth area.” We isolate this improved grounding by masking class names from both full sentence descriptors and few-shot descriptors and found that 4-shot prompting accuracy goes from 34.6%, 30.1%, and 28.2% to 38.3%, 35.2%, and 33% for class, order, and family levels. This is not the full story, however. If we instead augment full sentence descriptors with common species names when available, we get 39.5%, 38.3%, and 35.4%. To summarize, our analysis shows much of the performance gains seen in full-sentence prompting for everyday item classification tasks can be attributed to class name leakage. Meanwhile, for the species domain, we see that class name leakage does not have as much of an effect unless common species names are used, and instead few-shot prompting improves performance by producing better visually grounded descriptors for long-tailed classes. We will update with this analysis in the revision.

  4. Please see overall response #2 titled “MLLM confounding factors.”

Questions:

  1. BioCLIP-2 is an interesting case. Specifically, the model was trained to be a foundational model for species classification. Therefore, it was fed a bunch of images paired with taxonomic data. This supervised learning seems to have led to a certain level of catastrophic forgetting of the base VLM’s capabilities. Thus, when the model is provided with class names with visual descriptors, we are forcing the model to remember pre-training capabilities, which is always difficult when such a strong distribution shift is introduced at the fine-tuning stage.

  2. Since sentence visual descriptors need a subject to follow standard grammatical rules, we find that indeed the sentence length descriptors are more likely to include entry-level categories than the standard CbD prompting. Specifically, on the six everyday item domains, we see the frequency of classes with entry-level categories in their descriptors rises from 7.7% to 19% and 17% to 56% for intermediate and fine tasks respectively. The species domain also shows an increase in frequency from 50% to 58%, 47% to 51%, and 46% to 51%. The increase is more subtle here because even standard CbD descriptors tend to include organism types and, thus, the longer sentence descriptors deviate less strongly.

Dear Reviewer Fzn3,

Thank you again for your review. With the discussion period closing on November 18th, we wanted to check whether our response sufficiently addressed your concerns. In particular, we hope our analysis of the prompting strategy ablation experiment provided sufficient depth. Moreover, we believe our additional ablations and associated discussion provide further insights into our claims about pre-training datasets and MLLM inference confounding factors.

If anything is still unclear, we would be glad to clarify before the deadline. Thank you for your time!

Thank you for the detailed responses and additional analyses. While I appreciate the authors' efforts to address my concerns, I believe the central concern regarding the disentanglement of the recognition bottleneck from descriptor quality remains insufficiently addressed.

Therefore, I will maintain my original rating (Weak Reject).

We did a human study on a 10-task subset (49 classes) of TaxoBench to more closely disentangle descriptor quality from VLM recognition performance. We found visual discriminative (aka "useful") descriptors were harder for LLMs to generate. However, when filtering for this metric, we found a sizable boost in fine-grained recognition performance (24.90 to 45.37). Please see details in the third point of Follow-Up to Overall Response to Reviewers with Additional Experiments.

We hope this sufficiently addresses your central concern about disentangling the recognition bottleneck from descriptor quality and will include a larger human study to further address this concern in the camera-ready version. We would be happy to address any follow-up questions.

Reviewer_FQvAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether LLM-generated visual descriptions genuinely improve zero-shot image classification. To answer this question, this paper introduces TaxoBench, a new benchmark covering seven real-world domains and multiple levels of classification granularity. Through extensive experiments, the results show that Classify by Descriptions (CbD) provides limited improvements over class-name-based classification and that its performance drops substantially when class names are removed, especially in fine-grained settings. The paper further investigates two potential bottlenecks: the quality of LLM-generated descriptions and the ability of VLMs to recognize these descriptions. Several mitigation strategies are evaluated, including descriptor optimization, prompting strategies, and stronger recognition models. While some improvements are observed, the results highlight the limitations of current CbD methods in reliable and interpretable visual classification.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The paper investigates an interesting and important question: whether the improvements from LLM-generated descriptions in VLM classification actually come from better recognition of visual details, rather than class-name information.

  3. TaxoBench is a meaningful benchmark. By considering both domain diversity and classification granularity, it provides a more comprehensive evaluation than existing benchmarks that focus on a limited set of fine-grained classification tasks.

Critical weaknesses
  1. The reliance of CLIP-like model classification on class names, rather than the additional information in generated descriptions, has already been investigated in prior works, such as WaffleCLIP [1]. The paper would benefit from clarifying what new insights its analysis provides beyond these findings.

  2. Removing class names may also remove useful category information for classification. Different classes can share some same attributes, so recognizing these attributes alone may not be sufficient for classification. Furthermore, the examined CbD method simply averages descriptor scores without considering their importance or correlations, potentially weakening the contribution of discriminative attributes.

  3. Although the paper identifies descriptor generation and descriptor recognition as two main bottlenecks, most analyses investigate them indirectly by changing prompts or models. The appendix includes an attribute recognition experiment with human annotations, but the paper does not systematically quantify how much each bottleneck contributes to classification errors. This limits the insights that the analysis can provide for future research.

  4. The improvements from mLLMs may not solely result from stronger visual attribute recognition, since the models also use different scoring and prompting procedures. Additional controlled comparisons would help clarify the source of these improvements.

[1] Roth, Karsten, et al. "Waffling around for performance: Visual classification with random words and broad concepts." ICCV. IEEE, 2023.

Questions for authors
  1. Can the authors better distinguish the contribution of visual descriptions from the contribution of class-name information, perhaps using alternative descriptor aggregation methods or controlled descriptor baselines?

  2. Can the authors provide a more direct analysis of the relative contributions of descriptor quality and recognition accuracy to overall classification performance?

3 comments
AuthorsOpenReview ↗

Review Response

We hope to address your comments and concerns below.

Weaknesses:

  1. We agree that class name removal and even entry-level categorization have been previously explored. While these ideas have been explored, TaxoBench lets us test how pervasive these claims are across application domains and recognition granularities. Moreover, our analysis also includes the following novel conclusions: (1) CbD does not provide a substantial benefit over standard zero-shot and few-shot classification practices even in fine-grained settings; (2) the language bottleneck of generating visual descriptors can be improved either with more verbose, and thus less interpretable, descriptors and few-shot examples; (3) entry-level categorization seems to be a happy medium to get some of the benefits of more verbose descriptors without sacrificing interpretability; and (4) MLLMs improve over CLIP-style VLMs for recognition of descriptors but incur significant computational costs. Ways to bridge CLIP models’ efficient inference and MLLMs’ performance show some promise, but further research is needed for practical use.

  2. We agree that simple removal of class names can be an overly restrictive inference setting. In fact, that observation motivated our exploration of entry-level categorization (Section 5.3). To provide further illumination, we re-ran the experiment with super categories (i.e., instead of “bird” using “animal” or instead of “chair” using “a piece of furniture”). We found that, even with these objectively less informative categories, the performance only drops from 38.95 to 38.87 and 18.09 to 17.93 for the easy and hard tasks respectively. This emphasizes the importance of instilling this context into visual descriptors and how little context is needed for CbD to benefit from it.

As for the second point regarding the simplistic nature of averaging image-descriptor matching scores into class scores to perform classification, we agree that many other options are possible. For example, many prior works, such as LaBo [1], LM4CV [2], and CDL [3], learn a mapping from per-descriptor scores to class scores. However, this is outside the scope of our paper, which focuses on CbD methods that do not assume availability of training data to learn such a mapping. Nevertheless, we agree that approximating how much performance could be lost at this aggregation step is helpful. Therefore, as an upper bound, we prompted GPT-4.1 to directly output a class ID given an image and “class ID”-organized visual descriptors. Under this setup, we see performance surges from 52.57% and 33.28% to 77.26% and 58.33% respectively. This hints at a potential performance gap, but further analysis is needed to properly isolate the aggregation step.

  1. Please see overall response #3 titled “Disentangling VLM recognition performance from LLM-generated descriptor quality.”

  2. Please see overall response #2 titled “MLLM confounding factors.”

Questions:

  1. We agree that distinguishing the contributions of visual descriptions and class-name information is important. We examine this through several controlled descriptor baselines, including ablations with and without prepending the class name to the descriptors in Table 2. Holding the descriptors fixed, this comparison isolates the incremental benefit of explicitly including the class name alongside the visual descriptions.

We also investigate whether providing additional class-name information during descriptor generation improves performance. Specifically, we keep the prompt wording unchanged but add the names of the classes being distinguished. This discriminative prompting strategy reduces performance as seen in the table below, indicating that providing competing class names during generation does not improve the resulting descriptors in our setting. Clearly, this additional controlled descriptor baseline sheds additional light on the limitations of what can and can not help the descriptor generation process. A more comprehensive analysis of this and other controlled ablations for prompting strategies will be included in the revised version.

CategoryTask DifficultyCbDIn-Context Distinctive Prompting
SpeciesClass16.469.30
Order13.538.22
Family8.177.02
Everyday ItemsIntermediate35.3527.25
Fine14.7512.48
  1. Please see overall response #3 titled “Disentangling VLM recognition performance from LLM-generated descriptor quality.”

[1] Yang, Yue, et al. "Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification." CVPR. IEEE, 2023.

[2] Yan, An, et al. "Learning Concise and Descriptive Attributes for Visual Recognition." ICCV. IEEE, 2023.

[3] Zhang, Yuan, et al. "Pre-trained Vision-Language Models Learn Discoverable Visual Concepts." TMLR. 2025.

Dear Reviewer FQvA,

Thank you again for your review. With the discussion period closing on November 18th, we wanted to check whether our response sufficiently addressed your concerns. In particular, we hope our delineation of re-established versus novel claims is clearer. Moreover, we believe our additional ablations and associated discussion provide further insights into the impact of semantic context, what can be expected from controlled descriptor baselines, and how much the batched scoring setup confounds MLLM performance gains.

If anything is still unclear, we would be glad to clarify before the deadline. Thank you for your time!

Thank you for the detailed responses. I will retain my positive score.

Reviewer_7U9jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper critically investigates the effectiveness of Classify by Descriptions (CbD), a zero-shot image classification paradigm in which an LLM generates class-specific visual descriptions and a vision-language model evaluates their compatibility with an input image. The authors introduce TaxoBench, a benchmark spanning seven classification domains and multiple levels of category granularity, with the aim of assessing CbD under more realistic and fine-grained recognition settings. The paper compares CbD with standard zero-shot and few-shot classifiers, examines the effects of including class names in descriptions, investigates alternative descriptor-generation strategies, and evaluates multiple vision-language architectures, including multimodal LLMs. The authors also explore a patch-aware image-text matching architecture. The principal findings are that CbD frequently relies on class-name information, achieves limited performance in difficult recognition settings, and remains constrained by both the quality of generated descriptions and the ability of vision-language models to recognize them.

Critical strengths
  1. A useful benchmark with broader domain and granularity coverage. TaxoBench combines hierarchical classification tasks across biological species and everyday-object domains. The inclusion of multiple levels of category granularity makes it possible to investigate how recognition behavior changes as candidate classes become more semantically related. The benchmark construction is described in substantial detail in Appendix B, including data selection, hierarchy construction, and sampling procedures. This is a meaningful contribution because evaluation design can strongly influence conclusions about fine-grained visual recognition.

  2. A broad empirical examination of the CbD pipeline. The paper examines several relevant factors, including class-name inclusion, alternative language prompting strategies, descriptor optimization, and different recognition models. The comparison between CLIP-style models and multimodal LLMs is particularly useful for identifying potential directions for improving description-based classification. The additional experiments on discriminative versus representative descriptions (Appendix F.1) and human-annotated attributes (Appendix F.2) further strengthen the empirical scope of the work.

Critical weaknesses
  1. The main experiments do not establish whether the degradation under fine-grained classification is particularly severe for CbD. A central motivation is that CbD becomes unreliable as classification requires finer-grained distinctions. However, the reported results also show substantial performance degradation for conventional zero-shot classification, and the relative changes are remarkably similar. For example, in Table 2, using CLIP: Standard zero-shot classification decreases from 63.15 on easy tasks to 36.99 on hard tasks, a drop of 26.16 percentage points. CbD with class names decreases from 67.57 to 41.20, a drop of 26.37 percentage points. The same pattern is observed with OpenCLIP: Standard zero-shot classification decreases from 69.22 to 45.94, a drop of 23.28 points. CbD with class names decreases from 72.34 to 49.36, a drop of 22.98 points. Moreover, CbD with class names maintains a positive improvement over standard zero-shot classification in both difficulty settings. Therefore, the reported results establish that fine-grained recognition is difficult, including for CbD, but they do not establish that increasing granularity disproportionately harms CbD compared with conventional zero-shot recognition. This distinction is important because difficulty under fine-grained recognition is not necessarily a CbD-specific failure. A stronger analysis would compare the relative advantage of CbD over matched baselines as granularity increases, preferably within each domain and with uncertainty estimates. The novelty of the diagnostic conclusion also deserves clearer positioning. Section 2 already acknowledges prior findings by Baron et al. (2024) and Roth et al. (2023) concerning the reliance of description-based classification on class names and limitations of visual attribute recognition. The new benchmark may offer important evidence across additional settings, but the paper should distinguish previously established limitations from genuinely new empirical findings.

  2. The experiments do not fully isolate the proposed causes of CbD failure, making some of the causal interpretations stronger than the evidence supports. The paper identifies two principal bottlenecks: inaccurate LLM-generated descriptions and unreliable VLM recognition of those descriptions. These are plausible and important hypotheses. However, end-to-end classification accuracy alone cannot reliably separate them. First, the significant drop after removing class names (Section 4, Table 2) demonstrates that class-name information is important to the current implementation. It does not, by itself, demonstrate that the generated descriptions are not interpretable or that the model is ignoring their semantic content. Removing the class name also changes the linguistic context supplied to a model pretrained primarily on image-caption associations. A description may be visually meaningful but difficult to interpret without an object or category reference. Thus, dependence on class-name context and the faithfulness of description-based explanations are related but distinct questions. The entry-level categorization experiment in Section 5.3 partially addresses the importance of semantic context. However, it does not directly quantify how much the predicted class depends on correct visual attributes versus category-name information. Second, experiments that improve classification by changing descriptors or recognition models do not fully identify the relative contributions of descriptor errors and recognition errors. A stronger decomposition would require independently checking whether the generated attributes are correct and visually observable, and then testing whether the recognition model detects those verified attributes. The human-annotated CUB attribute experiment in Appendix F.2 is relevant and should be acknowledged, but it is conducted on a different attribute dataset and does not directly establish the accuracy of the generated descriptions used in TaxoBench. Finally, the interpretation involving model pretraining coverage is not fully isolated. Section 4 describes species recognition as a setting involving categories likely to be unseen during CLIP pretraining, and interprets differences between CLIP and BioCLIP 2 as evidence concerning pretraining exposure. However, actual class- and image-level pretraining overlap is not measured in the reported analysis. Consequently, the evidence establishes several important empirical sensitivities of CbD, but it does not yet justify all the stronger conclusions concerning lack of interpretability, insufficient descriptor quality, and pretraining coverage as distinct causes.

Questions for authors

Q1. Is fine-grained degradation disproportionately severe for CbD? Given the nearly identical easy-to-hard accuracy decreases of standard zero-shot classification and class-name-conditioned CbD in Table 2, can the authors report a controlled analysis of the relative CbD improvement over the corresponding zero-shot baseline across granularity levels? This would clarify whether TaxoBench reveals a distinctive limitation of CbD or a more general limitation of the underlying recognition models.

Q2. Can descriptor correctness and recognition accuracy be measured independently on TaxoBench? Could the authors manually verify a representative sample of generated descriptors for factual correctness, visual observability, and discriminative usefulness? Evaluating VLM recognition on those verified descriptors would help determine whether failure primarily originates from the language-generation stage or the visual-recognition stage. Appendix F.2 provides a related experiment using CUB annotations, but not a direct decomposition of errors on the proposed benchmark.

Q3. How is pretraining exposure controlled? The species benchmark is constructed from TreeOfLife-200M, while BioCLIP 2 is a model developed using closely related biological training resources. Have the authors checked whether the evaluation species or individual images were present in BioCLIP 2's training data? More generally, how are the claims about familiar versus unfamiliar categories supported without measurements of pretraining exposure? A stratified analysis would help distinguish genuine generalization from recognition of previously encountered categories.

Q4. How much of the mLLM improvement is attributable to the model versus the inference procedure? Appendix E.2 explains that GPT-4.1 and Qwen-3-VL use different descriptor grouping and prompting procedures, and that the mLLM experiments use a single task-sampling seed. Since the study emphasizes the improvement from using mLLMs as recognition models, could the authors provide uncertainty estimates and clarify how much of the gain is attributable to the model choice rather than differences in the scoring interface and prompt context?

2 comments
AuthorsOpenReview ↗

Review Response

We thank you for your insightful feedback and hope to address your comments and concerns below.

Response to Weakness 1:

W1(a): Discussion of fine-grained recognition results

The main experiments do not establish whether the degradation under fine-grained classification is particularly severe for CbD. A central motivation is that CbD becomes unreliable as classification requires finer-grained distinctions… the reported results establish that fine-grained recognition is difficult, including for CbD, but they do not establish that increasing granularity disproportionately harms CbD…

We agree that our results do not support the claim that CbD is poorer at fine-grained recognition than classic zero-shot classification. Instead, our claim in Section 4 is that, intuitively, the additional information from the LLM-generated descriptors should help CbD improve over zero-shot classification when distinguishing fine-grained categories (Line 258-262), but this does not happen: Surprisingly, CbD still struggles in the same way as zero-shot classification.

W1(b): Previously established limitations versus new findings

The novelty of the diagnostic conclusion also deserves clearer positioning….but the paper should distinguish previously established limitations from genuinely new empirical findings.

We would like to re-iterate the novel claims posed in our analysis: (1) CbD does not provide a substantial benefit over standard zero-shot and few-shot classification practices even in fine-grained settings; (2) the language bottleneck of generating visual descriptors can be improved either with more verbose, and thus less interpretable, descriptors and few-shot examples; (3) entry-level categorization seems to be a happy medium to get some of the benefits of more verbose descriptors without sacrificing interpretability; and (4) MLLMs improve over CLIP-style VLMs for recognition of descriptors but incur significant computational costs. Ways to bridge CLIP models’ efficient inference and MLLMs’ performance show some promise, but further research is needed for practical use. We will update the text to better delineate new claims with respect to prior work.

Response to Weakness 2:

W2(a): Disentangling VLM recognition performance from LLM-generated descriptor quality

The experiments do not fully isolate the proposed causes of CbD failure… The paper identifies two principal bottlenecks: inaccurate LLM-generated descriptions and unreliable VLM recognition of those descriptions… end-to-end classification accuracy alone cannot reliably separate them.

Please see our overall response #3 titled “Disentangling VLM recognition performance from LLM-generated descriptor quality.”

W2(b): Pre-training coverage of TaxoBench

Finally, the interpretation involving model pretraining coverage is not fully isolated… However, actual class- and image-level pretraining overlap is not measured in the reported analysis.

Please see our overall response #1 titled “Pre-training coverage analysis.”

Questions:

  1. Based on our results, we find that performance degradation is a common phenomenon across both traditional zero-shot classification and CbD. However, it is more surprising for CbD because fine-grained classification is precisely where we would expect descriptions to help.

  2. Please see overall response #3 titled “Disentangling VLM recognition performance from LLM-generated descriptor quality.”

  3. Pre-training exposure is an important consideration. In the specific case of BioCLIP-2, the model is exposed to the entire web-crawled corpus of iNaturalist species that our benchmark derives species classification tasks from. Therefore, BioCLIP-2, especially in the traditional zero-shot classification setting, represents the oracle, best-case scenario of supervised learning on that task. However, its subsequent loss of performance when incorporating natural language descriptors shows that difficulties remain for task-specific foundational models when practitioners want to move towards trustworthy recognition. See overall response #1 titled “Pre-training coverage analysis” for further discussion.

  4. Please see overall response #2 titled “MLLM confounding factors.”

Dear Reviewer 7U9j,

Thank you again for your review. With the discussion period closing on November 18th, we wanted to check whether our response sufficiently addressed your concerns. In particular, we hope our analysis of fine-grained results and delineation of re-established versus novel claims is clear. Moreover, our pre-training dataset analysis should adequately support our claims about the relative prevalence of species names versus common everyday items in VLM pre-training data mixes.

If anything is still unclear, we would be glad to clarify before the deadline. Thank you for your time!

Comments on the paper

3 comments
AuthorsOpenReview ↗

Overall Response to Reviewers

We thank the reviewers for their time and thoughtful feedback. Here, we have answers to questions commonly raised by the reviewers. We address each reviewer’s individual concerns in the comments below.

  1. Pre-training coverage analysis Reviewers 7U9j and Fzn3 question our assertion that a major factor in VLM’s reliable recognition is how often the respective class names appear in pre-training. We made this assertion based on the commonly held belief that scientific data (e.g., scientific species names) are less commonly represented than typical everyday nouns. However, we agree that some supportive quantitative evidence is necessary.

Using the methodology proposed in Udandarao, Vishaal et al. [1], we measured the frequency of class names from TaxoBench in an open-source pre-training dataset (LAION-400M [2]). In the LAION-400M dataset, we find species class names show up roughly 13.5% of the time, while the clothing, furniture, groceries, kitchenware & tableware, musical instruments, and tools show up 100%, 100%, 54.6%, 96.6%, 100%, and 82.8% of the time respectively. Clearly, species classification is underrepresented in common pre-training datasets. We will include a more comprehensive analysis of this result and an equivalent one on the LAION-2B dataset [3] in the revised version.

  1. MLLM confounding factors Reviewers 7U9j, FQvA, and Fzn3 note that our claim that MLLMs improve over standard CLIP-style models for open-vocabulary attribute recognition is confounded by the differing prompting setups of MLLM and CLIP inference. To explore how much of a role our cost-saving MLLM inference setup plays, we sampled a diverse subset of 6 tasks where we saw significant performance gains when using GPT-4.1 and ran a new inference protocol that mimics the context and scoring CLIP models perform.

Specifically, we individually score image-descriptor pairs on a yes/no basis and calculate normalized confidence scores using the log probabilities of the “yes” and “no” output tokens. From the table below, using the chance-adjusted macro accuracy metric, we see performance drops when moving from the batched scoring setup, but still significant improvements over CLIP are observed. We will perform a larger-scale experiment investigating the extent of this observation using Qwen-3-VL in the revision.

DomainDifficultyTaskCLIPGPT-4.1 (old)GPT-4.1 (new)
ClothingEasy (Intermediate)Full-Body Garments3662.782.7
ClothingHard (Fine)Sweaters1.06040
Kitchenware & TablewareEasy (Intermediate)Cooking289278
Kitchenware & TablewareHard (Fine)Shallow Cookware166652
ToolsEasy (Intermediate)Shaping and finishing tools-4.09276
ToolsHard (Fine)Mallets2.07468
OverallEasy2082.2378.90
OverallHard6.3366.6753.33
  1. Disentangling VLM recognition performance from LLM-generated descriptor quality Reviewers 7U9j, FQvA, and Fzn3 had concerns about the degree to which our analysis can determine whether incorrect recognition of descriptors or poor descriptor quality caused a particular misclassification. To address this, we are running a human study on a uniformly sampled subset of TaxoBench’s everyday item classification tasks that requires participants to provide confidence scores on LLM-generated descriptors and generate descriptors themselves. We can then see how sensitive recognition performance is to the quality of descriptors. Moreover, these descriptors can test how varying the VLM affects performance under vetted, high-quality descriptors. We will put the results and discussion in the revised version.

Thank you all for your time, and we are happy to answer any follow-up questions.

[1] Udandarao, Vishaal, et al. "Pretraining Concept Frequency determines Multimodal Model Performance." NeurIPS. 2024.

[2] Schuhmann, Christoph, et al. "Laion-400m: Open dataset of clip-filtered 400 million image-text pairs." NeurIPS Data-Centric AI Workshop. 2021

[3] Schuhmann, Christoph, et al. "LAION-5b: An open large-scale dataset for training next generation image-text models" NeurIPS. 2022.

AuthorsOpenReview ↗

Follow-Up to Overall Response to Reviewers with Additional Experiments

Here, we have included follow up experiments to the main concerns raised by reviewers.

  1. Pre-training coverage analysis. We analyzed the larger LAION-2B pre-training dataset as proposed. Due to time constraints, we processed a random 12.5% subset of the dataset and found similar results to LAION-400M. Specifically, we find that species class names show up roughly 11.1% of the time, while the clothing, furniture, groceries, kitchenware & tableware, musical instruments, and tools show up 100%, 100%, 53.8%, 96.6%, 96.6%, and 81.1% of the time respectively. We will include the full dataset analysis in the camera-ready version.

  2. MLLM confounding factors. We claim CbD’s recognition bottleneck is alleviated with improved recognition capabilities of MLLMs (Line 462-464). However, since MLLM’s score descriptors in batches while CLIP only “sees” one descriptor at a time, reviewers FQvA and Fzn3 argue that our MLLM performance gains could hinge on the available context at inference time rather than better descriptor recognition. To address this concern, we ran a five-seed experiment over the six everyday item domains in TaxoBench (i.e., clothing, furniture, groceries, kitchenware & tableware, musical instruments, and tools). We then independently scored descriptors with Qwen-3-VL-32B using a yes/no log probability score to mimic CLIP inference. We found the Qwen model still significantly improves over CLIP, achieving 58.48 and 37.18 over 35.35 and 14.75 on the intermediate and fine settings respectively. This emphasizes the sizable gap between CLIP-style and MLLM open-vocabulary attribute recognition capabilities.

  3. Disentangling VLM recognition performance from LLM-generated descriptor quality. We conducted a pilot study with three human annotators on 10 tasks spanning furniture, groceries, kitchenware & tableware, musical instruments, to provide some insight into disentangling recognition from descriptor quality. To accomplish this, we first asked annotators to characterize the overall quality of LLM-generated descriptors using three 1-5 metrics for factual correctness, visual observability, and discriminative usefulness as per reviewer 7U9j comment. We found that people generally found descriptors to be factually correct and visually observable, but sometimes not discriminatively useful (avg score 2.6 out of 5). When we then sort and top-k filter descriptors based on usefulness, we see fine-grained recognition improves from 24.90 to 45.37 using the same CLIP model for scoring: signalling descriptor quality is a bottleneck alongside VLM recognition. We will include a larger study across TaxoBench tasks and more in-depth analysis in the camera-ready version.

AuthorsOpenReview ↗

Rebuttal Summary

We would like to thank the reviewers for taking the time to read our paper and provide insightful feedback. Below we summarize the main concerns raised and our responses.

  1. Re-established vs novel claims (7U9j, FQvA): We highlight four new claims: (1) CbD does not provide a substantial benefit over standard zero-shot and few-shot classification practices even in fine-grained settings; (2) the language bottleneck of generating visual descriptors can be improved either with more verbose, and thus less interpretable, descriptors and few-shot examples; (3) entry-level categorization seems to be a happy medium to get some of the benefits of more verbose descriptors without sacrificing interpretability; and (4) MLLMs improve over CLIP-style VLMs for recognition of descriptors but incur significant computational costs.

  2. Disentangling VLM recognition from LLM-generated descriptor quality (7U9j, FQvA, Fzn3): We provide a pilot study showing the importance of descriptor quality and how it affects recognition performance (see point #3 in the overall response and follow-up overall response).

  3. Pre-training coverage analysis (7U9j, Fzn3): We provide quantitative evidence on LAION-400M and a random subset of LAION-2B for our claim that species classification is a relatively unseen domain when compared with the other six everyday item domains in TaxoBench (see point #1 in the overall response and follow-up overall response).

  4. Class name removal and CbD score aggregation hurts performance (FQvA): Our entry-level categorization experiments highlight the need to not just remove class information but rather replace it with a shared semantic context. We perform a follow-up ablation to show even a super-category/very coarse categorization can help. We then run a MLLM experiment to establish an approximate upper bound on score aggregation performance gains for our benchmark.

  5. MLLM confounding factors (FQvA, Fzn3): We substantiate our claim about MLLM’s superior recognition capabilities by switching the batched scoring regime with a per-descriptor one and finding MLLMs still better recognize descriptors than CLIP-style models (see point #2 in the overall response and follow-up overall response).

  6. Deeper analysis of prompting strategy results (Fzn3): We perform an in-depth analysis on why species classification improves with few-shot prompting; while, everyday item domains more so improve with the longer visual descriptors. We find the class leakage in longer descriptors is more helpful to everyday item domains whereas few-shot prompting can fix some of the descriptor generation mistakes on rare species.

We address initial concerns for all reviewers. Reviewer Fzn3 raised a follow-up concern with the disentangling of descriptor quality and VLM recognition performance which we believe is addressed with our human study. Moreover, reviewer Fzn3’s informativeness score of 3 shows the work’s novelty and contribution to research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.