- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors are motivated by the unreliability of current benchmarks for Classify by Descriptions (CbD). Current benchmarks feature limited domain coverage and limited coarse-to-fine recognition coverage. To address this problem, the paper introduces TaxoBench, providing broad coverage of realistic classification problems, common and uncommon category names, and both coarse- and fine-grained recognition problems. TaxoBench is used to pose and answer three important questions. Evaluation with TaxoBench shows that current approaches struggle with both producing good descriptions and recognizing these descriptions. Comparison between CbD and black-box methods confirms that zero-shot performance relies on the overlap between unseen classes and pre-training data, and that descriptions are not useful even in fine-grained scenarios. Further analysis identifies two major bottlenecks: description generation and recognition ability. Different prompting strategies and entry-level categorization improve descriptor quality, while existing optimization methods show limited gains. In terms of recognition ability, more powerful recent CLIP-based models provide modest improvements, while mLLMs achieve substantial gains. Overall, the results highlight the limitations of current CbD approaches and the need for better description generation and recognition.
Critical strengths
-
The paper introduces a new benchmark, TaxoBench. The motivation behind TaxoBench and the research questions are clear, well-justified, and interesting. The paper clearly identifies three key limitations of CbD: 1) Reliance on class names, 2) Language bottleneck, and 3) Recognition bottleneck.
-
TaxoBench covers diverse real-world domains and multiple levels of classification granularity, enabling a more comprehensive evaluation of CbD across different domains and difficulty levels.
-
The paper presents a well-designed experimental setup and a thorough analysis of the results, providing valuable insights into the fundamental limitations of both description generation and recognition in CbD.
Critical weaknesses
-
In Table 2, the effectiveness of CbD varies significantly across different models. While CLIP and OpenCLIP benefit from CbD, BioCLIP2 shows a substantial performance drop. Moreover, the contribution of descriptions appears limited compared to that of class names in most settings. These observations raise questions about whether the results sufficiently support the claimed recognition bottleneck. Can the recognition bottleneck be clearly disentangled from the low quality of generated descriptions? Although Section 6.1 compares different VLMs using the same descriptions, the recognition bottleneck is not sufficiently disentangled from the language bottleneck. Evaluating whether similar results are obtained with higher-quality descriptions would provide stronger evidence for the claimed recognition bottleneck.
-
In line 241, the authors suggest that one reason CbD relies heavily on class names is their familiarity from pre-training. However, VLMs such as CLIP are pre-trained on large-scale web-crawled image-text pairs, which may contain not only class names but also rich natural language descriptions. Therefore, it is unclear whether the observed behavior can be attributed specifically to class name familiarity without further analysis of the pre-training data.
-
The analysis of the two prompting strategies in Table 4 is limited. While full-sentence prompting largely improves performance on everyday item classification, the improvements from few-shot in-context prompting are more pronounced in species classification. However, according to the analysis in Section 4, species classification tasks particularly benefit from descriptions. Thus, despite full-sentence prompting providing more detailed descriptions, further analysis is needed to explain why it is less effective in species classification. A deeper analysis of these results would strengthen understanding of how description quality affects CbD performance.
-
In Section 6.2, it is unclear whether mLLMs truly address the recognition bottleneck or simply benefit from their rich general world knowledge. The limited improvements in Section 6.3 further raise questions about whether the gains from mLLMs can be attributed primarily to better descriptor recognition.
Questions for authors
-
Why does CbD with BioCLIP2 perform substantially worse than black-box zero-shot classification even when class names are included? While the performance degradation without class names is reasonably explained, the degradation with class names is not sufficiently investigated. Is this due to the language bottleneck or the recognition bottleneck?
-
Do full-sentence descriptions actually contain entry-level category information?
4 comments
Dear Reviewer Fzn3,
Thank you again for your review. With the discussion period closing on November 18th, we wanted to check whether our response sufficiently addressed your concerns. In particular, we hope our analysis of the prompting strategy ablation experiment provided sufficient depth. Moreover, we believe our additional ablations and associated discussion provide further insights into our claims about pre-training datasets and MLLM inference confounding factors.
If anything is still unclear, we would be glad to clarify before the deadline. Thank you for your time!
Thank you for the detailed responses and additional analyses. While I appreciate the authors' efforts to address my concerns, I believe the central concern regarding the disentanglement of the recognition bottleneck from descriptor quality remains insufficiently addressed.
Therefore, I will maintain my original rating (Weak Reject).
We did a human study on a 10-task subset (49 classes) of TaxoBench to more closely disentangle descriptor quality from VLM recognition performance. We found visual discriminative (aka "useful") descriptors were harder for LLMs to generate. However, when filtering for this metric, we found a sizable boost in fine-grained recognition performance (24.90 to 45.37). Please see details in the third point of Follow-Up to Overall Response to Reviewers with Additional Experiments.
We hope this sufficiently addresses your central concern about disentangling the recognition bottleneck from descriptor quality and will include a larger human study to further address this concern in the camera-ready version. We would be happy to address any follow-up questions.
Review Response
We thank you for your insightful feedback and hope to address your comments and concerns below.
Weaknesses:
Please see overall response #3 titled “Disentangling VLM recognition performance from LLM-generated descriptor quality.”
Please see overall response #1 titled “Pre-training coverage analysis.”
In our subsequent analysis, we found that full-sentence prompting mainly helps everyday item domains by reintroducing category names that CLIP knows well into descriptors. In fact, when we mask class names from descriptors and replace them with generic super categories, we find the overall performance on everyday item domains drops from 44.4% and 69.2% to 42.2% and 58.4% on fine and intermediate tasks, respectively. This is strictly less than the corresponding performance of masked descriptors from few-shot prompting (i.e., 42.9% and 59.5%). The species domain similarly introduces organism names. However, unlike the everyday item domains, these names are typically scientific abbreviations or Latin binomials (e.g., A. aequalis) which show up much less frequently in standard web-scraped internet text captions. In fact, when we masked out names from few-shot and full-sentence descriptors, performance dropped no more than 1 percentage point across class, order, and family levels. On the other hand, few-shot prompting helps ground descriptors, especially for long-tailed class names. For example, the sea anemone “Heteranthus verruculatus” initially was mistaken for a plant with descriptors like “The plant has a shrubby or small bush-like form with multiple stems.” and “Leaves are typically opposite, simple, and may have a slightly rough or textured surface.” When provided species observations in context, the produced descriptors properly identify sea anemone visual features like “Tentacles often have a warty or knobby appearance.” and “Tentacles arranged radially around the central mouth area.” We isolate this improved grounding by masking class names from both full sentence descriptors and few-shot descriptors and found that 4-shot prompting accuracy goes from 34.6%, 30.1%, and 28.2% to 38.3%, 35.2%, and 33% for class, order, and family levels. This is not the full story, however. If we instead augment full sentence descriptors with common species names when available, we get 39.5%, 38.3%, and 35.4%. To summarize, our analysis shows much of the performance gains seen in full-sentence prompting for everyday item classification tasks can be attributed to class name leakage. Meanwhile, for the species domain, we see that class name leakage does not have as much of an effect unless common species names are used, and instead few-shot prompting improves performance by producing better visually grounded descriptors for long-tailed classes. We will update with this analysis in the revision.
Please see overall response #2 titled “MLLM confounding factors.”
Questions:
BioCLIP-2 is an interesting case. Specifically, the model was trained to be a foundational model for species classification. Therefore, it was fed a bunch of images paired with taxonomic data. This supervised learning seems to have led to a certain level of catastrophic forgetting of the base VLM’s capabilities. Thus, when the model is provided with class names with visual descriptors, we are forcing the model to remember pre-training capabilities, which is always difficult when such a strong distribution shift is introduced at the fine-tuning stage.
Since sentence visual descriptors need a subject to follow standard grammatical rules, we find that indeed the sentence length descriptors are more likely to include entry-level categories than the standard CbD prompting. Specifically, on the six everyday item domains, we see the frequency of classes with entry-level categories in their descriptors rises from 7.7% to 19% and 17% to 56% for intermediate and fine tasks respectively. The species domain also shows an increase in frequency from 50% to 58%, 47% to 51%, and 46% to 51%. The increase is more subtle here because even standard CbD descriptors tend to include organism types and, thus, the longer sentence descriptors deviate less strongly.