acceptodds
Withdrawn from ICLR 2027

Neural Catalog: Scaling Species Recognition with Catalog of Life-Augmented Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling fine-grained recognition from curated benchmarks (200 species) to real-world inventories (11,202 species) costs vision-language models more than 30 points of accuracy, exposing a structural limitation: discriminating among thousands of visually similar candidates requires reasoning over relative differences rather than absolute matching. We introduce CoLAG (Catalogue of Life Augmented Generation), a training-free system built on a catalogue of 11,202 species indexed as discriminative summaries rather than raw articles. CoLAG pairs a multimodal retrieval stage with neighborhood-subtracted embeddings that amplify discriminative dimensions by suppressing features shared among confusable species, and a comparative reasoning stage that presents a multimodal LLM with retrieved candidates, discriminative descriptions, and calibrated retrieval scores for joint evaluation. Across five bird benchmarks, marine species, and Pok\'emon, CoLAG adds points on average over direct prompting across six MLLM backbones, and points over independent scoring.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Species Recognition, Open Vocabulary Recognition, MLLM, Species Discovery

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TmSQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposed CoLAG, a two-stage training-free method to improve the performance of open-vocabulary species recognition of MLLMs. CoLAG first retrieves relevant candidates, and then reasons over the retrieved candidates. The method was tested on five bird datasets and two other datasets (Pokemon and Fish) and improved the open-vocabulary species recognition of MLLMs on these datasets.

Critical strengths

None

Critical weaknesses
  • The problem is not well-motivated and the paper is poorly written. There is no discussion on why MLLMs are needed for species recognition, while the paper acknowledges that “standard MLLMs rely on implicit parametric knowledge and struggle to compare fine-grained attributes among visually similar species, causing confusion in open-vocabulary settings.” The fact that “the candidate set is large, evolving, and often unbounded” does not mean that species recognition needs to be treated as an open-vocabulary recognition problem. Why is text description needed to tackle this problem? If text description is necessary, this approach cannot be generalized to species without detailed descriptions, making this approach useless for those species. It is also unclear why species recognition should be treated as an “open-vocabulary” problem instead of an open-set recognition problem (to detect the species not in the training set) or a one-shot learning problem given that CoLAG requires an anchor image for each species. The paper does not study BioCLIP (Stevens et al., 2024), BioCLIP v2 (Gu et al., 2026), BioCAP (Zhang et al., 2026), or BioTrove-CLIP(Yang et al., 2024), foundation models trained for species recognition, but used general-purposed, much bigger MLLMs instead, making the solution unnecessarily complicated. The Pokemon dataset is also completely irrelevant to species recognition.

  • There is no discussion on BioCLIP, BioCLIP v2, BioCAP, or BioTrove-CLIP in the related work. Table 1 only includes results of CLIP, OpenCLIP, and SigLIP. None of the models was trained for species recognition. What’s the performance drop of the four CLIP-based species recognition models? Besides, in the result section, BioCLIP was evaluated for retrieval, but not for recognition. Why? On the contrary, BioCLIP was trained for specifically species recognition, not retrieval. Since CoLAG needs an anchor image for each species, what’s the performance of one-shot classification of BioCLIP, BioCLIP v2, or DINOv3 (Siméoni et al., 2025)? Or use these images for linear probing? All of these can serve as a strong baseline and should be evaluated before using CoLAG.

Stevens, Samuel, et al. "Bioclip: A vision foundation model for the tree of life." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Gu, Jianyang, et al. "Bioclip 2: Emergent properties from scaling hierarchical contrastive learning." Advances in Neural Information Processing Systems 38 (2026): 102778-102811.

Zhang, Ziheng, et al. "BioCAP: exploiting synthetic captions beyond labels in biological foundation models." International Conference on Learning Representations. Vol. 2026. 2026.

Yang, Chih-Hsuan, et al. "Biotrove: A large curated image dataset enabling ai for biodiversity." Advances in Neural Information Processing Systems 37 (2024): 102101-102120.

Siméoni, Oriane, et al. "Dinov3." arXiv preprint arXiv:2508.10104 (2025).

Questions for authors
  • Is there a description dataset for FishNet? If so, what was it curated? There is no description of this dataset
  • The paper is poorly written. For example, Section 4.1 is very unclear and difficult to read, especially the definition for the “chunks”. What is a “tower”? There’s no definition nor citation for “DINO”. Which version of DINO was used? “Congener” is a jargon. Use “within the same genus”.

Minor issues

  • Line 211 - there’s no definition of the signed generalized mean.
  • Section 4.1 - \Pi is often used as the product operator. Even though it is defined as the unit vector normalization in the paper, it is still very confusing.
  • Use \citep instead of \cite for in-line citations; it is very difficult to read now
  • Line 111 - 112: “species may be unseen, reclassified, or novel” - what’s the difference between unseen and novel?
  • Inconsistent font: CoLAG (italic) in Lines 65 and 74 vs CoLAG in the method section. Inconsistency in section 4.2 as well.
Reviewer_ubYGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoLAG, a catalogue-augmented framework for open-vocabulary fine-grained species recognition. The framework consists of two main components: CoLAGret, which combines multimodal embedding retrieval, DINOv2 visual reranking, hubness penalty, and symmetric neighborhood whitening to retrieve discriminative candidate species; and CoLAGgen, which enables a MLLM to compare the retrieved candidates using their visual descriptions and retrieval scores. The authors also construct a knowledge base covering 11,202 bird species, with concise, visually discriminative descriptions distilled from Wikipedia articles. Experiments on five bird benchmarks and two additional domains, FishNet and Pokémon, demonstrate improvements over direct MLLM prediction and candidate-wise scoring. Extensive ablations further investigate the contributions of individual retrieval components and the candidate comparison strategy.

Critical strengths
  1. The paper focuses on an important and challenging problem. Open-vocabulary fine-grained species recognition involves distinguishing among large numbers of visually similar categories, where conventional vision-language models can struggle. The proposed catalogue-augmented framework provides an interesting direction for incorporating external knowledge into fine-grained visual recognition.
  2. The framework combines retrieval and comparative reasoning in a reasonable way. The integration of multimodal retrieval, visual reranking, hubness penalty, and symmetric neighborhood whitening provides a structured approach to select candidate.
Critical weaknesses
  1. The paper evaluates recognition over a catalog of 11,202 bird species. The candidate space remains predefined. The experiments do not sufficiently examine how the framework behaves when the ground-truth species is absent from the catalog, or when none of the retrieved candidates is suitable. It remains unclear whether the framework can reliably handle ‘open world/vocabulary’ cases.
  2. The proposed framework combines multiple vision-language encoders, DINOv2-based reranking, and MLLM inference. However, the paper does not provide a sufficiently comprehensive evaluation of the end-to-end inference latency, throughput, and resource consumption of the complete pipeline. It is therefore difficult to assess the practical trade-off between recognition accuracy and computational cost.
  3. The evaluation primarily uses a bird catalog containing 11,202 species. It remains unclear whether the framework can maintain its retrieval quality and recognition accuracy when the candidate space expands to more species or encompasses diverse biological categories, such as animals and plants. The authors should also investigate how retrieval latency and computational costs scale with catalog size, as these factors may limit the framework's applicability to large-scale biodiversity databases.
Questions for authors
  1. Could the authors evaluate CoLAG under an open-set setting where some test species are deliberately excluded from the catalog, and report both known-species recognition accuracy and unknown-species detection performance?
  2. Could the authors provide a breakdown of the end-to-end inference latency and computational cost of each component, including the three vision-language encoders, DINOv2 reranking, and MLLM inference?
  3. How does CoLAG scale as the number of candidate species increases substantially beyond 11,202? Could the authors evaluate retrieval recall, final recognition accuracy, and inference latency at different catalog sizes?
  4. Could the authors evaluate CoLAG on a unified catalog containing diverse biological categories, such as birds, mammals, insects, and plants, rather than evaluating different domains separately? How does the increased diversity of visually similar categories affect retrieval and recognition performance?
Reviewer_Nh7yOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CoLAG (Catalog of Life Augmentation Generation), which is a training-free approach for species recognition. It uses a catalogue of species images and their textual descriptions and combines three different VLMs (CLIP, OpenCLIP, and SigLIP) to retrieve candidate species. CoLAG then refines the ranks of these candidate species using visual similarity and neighbourhood corrections. It then asks an MLLM to select a species from the retrieved candidates. They also construct the Birds and Pokemon datasets used for the inference. They show experiments on three different datasets birds, Pokemon and fishes.

Critical strengths
  • The paper addresses a known problem offine-grained species recognition and proposes an interesting training-free approach while utilising multiple VLM embeddings and DINO-based ranking for improving model predictions.
  • The catalogue construction is nice and combines Wikipedia visual features and only needs a few reference images.
  • The model ablations of using different VLMs are interesting and show how general Wikipedia texts degrade performance compared to distilled textual descriptions focusing only on visual traits.
  • The authors also discuss the upper bounds of the prediction accuracies where the retrieval errors occur when candidate species don't enter the candidate set.
Critical weaknesses
  • The paper in the introduction motivates the method with the need to solve reclassified and novel species with the reasoning capabilities and an evolving catalogue. The experiments do not evaluate whether the catalogue updates or whether we query species that are not present in the catalogue. Whether a catalogue with 11k species is enough and works for open-vocabulary recognition; if not, the paper should temper the claims.

  • The paper would benefit from an expanded related works section. The authors should discuss works like VLM4Bio [A], which also examines species recognition in an open-world setting, INQUIRE [B] and TARA [C] that present biological retrieval and recognition approaches.

  • L208 introduces 73k text-visual chunks for 11k species, but their construction is not fully explained. It is not clear how each species summary is divided into passages and how multiple reference images are being represented during the fusion. Also, do chunk-level ranking
    become a list of different species.

  • The neighbourhood corrections would benefit from a stronger comparison with existing retrieval corrections. Reweighting and one-sided control are nice, but does not fully establish its advantage over the hubness corrections. Should we compare these methods with the same embeddings?

  • Proposition one has narrow support in the conclusion than what was discussed. It assumes that two species will have identical neighbourhood centroids and also have equal projections onto the same centroid. The paper does not entirely show how closely the real species that can be confused satisfy these conditions. It does not generally show that a database-based correction cannot change the ranks.

  • Tables 12 and 13 evaluate at their own best β. It does not specify which validation data was used to select it, making the parameter selection unclear. It would be clearer if we mention the selection of neighbourhood sizes, the fusion parameters, correction weights, etc.

  • When we jointly present candidates, it improves accuracy, but it is not clear that how much of this improvement is coming from comparing fine-grained visual attributes and what is coming when we avoid the comparisons between independently generated scores. We can test the effect when we remove distinguishing attributes in the descriptions to support this.

  • The evaluation datasets do not seem to be complete. Sec 5.1 and L304 report 2443 Birdnsap test images, but Table 11 says 1851. For the iNaturalist dataset, it contains 1323 species with 13230 images in the validation set, but the official iNat-2021 validation subset for birds contains 1486 species with 14860 images. What was the filtering procedure, and why were a few species left out?

  • The paper is missing strong baseline comparisons. It should also report the stronger BioCLIP 2 [D] and BioCAP [E] for species-classification comparisons, and BioCAP also forms a visual Wikipedia text-driven classification model.

  • Similarly, the biologically trained VLM encoder is not fully addressed as a backbone choice for CoLAG. Tables 16 and 17 do report BioCLIP results with fused text-image representations, but it does not clearly compare the complete CoLAG pipeline using BioCLIP as an encoder choice against the same pipeline that uses CLIP, OpenCLIP and SigLIP. It would be interesting to replace the encoder with these stronger BioCLIP2/BioCAP models in the CoLAG pipeline.

  • Table 7 progressive ablations are interesting, and it shows the gains with the addition of each component. Although this is great, but it does not fully represent that all three encoders are necessary and disentangle the contribution of each. Maybe a combination of 2 of the three encoders has similar gains to using all three. Also, baseline results with a single encoder with all CoLAG-introduced components would be interesting to see.

References:

[A] Maruf, M., et al. "Vlm4bio: A benchmark dataset to evaluate pretrained vision-language models for trait discovery from biological images." Advances in Neural Information Processing Systems 37 (2024): 131035-131071.

[B] Vendrow, Edward, et al. "INQUIRE: A natural world text-to-image retrieval benchmark." Advances in Neural Information Processing Systems 37 (2024): 126500-126514.

[C] Hulingxiao He, Zhi Tan, Yuxin Peng; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 31124-31134

[D] Gu, Jianyang, et al. "Bioclip 2: Emergent properties from scaling hierarchical contrastive learning." Advances in Neural Information Processing Systems 38 (2026): 102778-102811.

[E] Zhang, Ziheng, et al. "BioCAP: exploiting synthetic captions beyond labels in biological foundation models." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  • How does the model perform with an increase in reference images for each species?
  • Why not use datasets like BIOCAP as a catalogue that have image-text pairs grounded in Wikipedia for 10M images and over 100k species?
  • In ablating the multiple-choice in Table 8, it shows a significant difference when returning an option letter versus returning the name of the species. Why does this happen when both use the same five candidates? Does the difference persist with a change in candidate order? Are the prompts different, or could the input prompting be responsible for this difference?
  • How does CoLAG convert retrieval scores into confidence values that are input to the MLLM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.