- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PhyRC (Phylogeny-Constrained Relative Competition), a new post-training method designed to improve taxonomic consistency and hierarchical visual recognition in Large Multimodal Models (LMMs). Although not novel (addressed in CLIP-like models), the central contribution is moving from category-based supervision to relationship-based supervision.
Critical strengths
The paper presents an extensive evaluation that goes beyond conventional accuracy metrics (i.e., hierarchcial consistency accuracy, leaf accuracy), assessing model robustness and taxonomic consistency under various training strategies and evaluation settings. This provides a more comprehensive understanding of model reliability and generalization, an this usually is not addresed in the emerging LMM benchmark paper courpus.
Critical weaknesses
The paper does not explicitly discuss its limitations. I encourage the authors to include a dedicated limitations section addressing the restricted taxonomic coverage, the focus on generative multimodal models, and the generalizability of the findings to other model architectures and more challenging biodiversity scenarios.
The dataset composition is not sufficiently clear, particularly regarding the number of images per species and the distribution of samples across taxonomic groups (~4.2K + 5.3k leaf categories, but over how many image samples? were they balanced). Given the well known class imbalance typically observed in biodiversity datasets, it would also be valuable to investigate whether the reported improvements in taxonomic consistency hold for underrepresented species, rather than being driven primarily by well-represented taxa.
The evaluation covers several open-weight multimodal models, including InternVL, Gemma, LLaVA, and Qwen. However, it would be valuable to extend the evaluation to proprietary models, such as GPT, Gemini, and Claude, to provide a broader comparison and assess whether the observed taxonomic consistency issues also occur in closed-source models.
Also, the table on page 8 is not corss-referenced.
Questions for authors
1.Why did the authors not consider CLIP-based models in their evaluation? Several strong biodiversity-specific vision-language models, such as BioCLIP and BioCLIP2, have demonstrated strong performance in taxonomic classification. Including these models would provide a valuable comparison and help assess whether the observed taxonomic consistency issues also extend to contrastive vision-language architectures.
2.Have the authors considered evaluating the models visually similar and difficult to distinguish species (i.e., Crypticbio[1])? Such an evaluation could provide further insights into the models' ability to maintain taxonomic consistency when fine-grained visual differences are limited.
3.Do models such as GPT, Gemini, and Claude exhibit similar taxonomic consistency problems?
[1] Manolache, Georgiana, Gerard Schouten, and Joaquin Vanschoren. "Crypticbio: A large multimodal dataset for visually confusing species." Advances in Neural Information Processing Systems 38 (2026).
7 comments
(2/4)Author rebuttal
Q1: Hierarchical Prediction Inconsistency in Contrastive Vision-Language Models
We thank the reviewer for this valuable suggestion. Our additional experiments reveal that hierarchical prediction inconsistency also exists in contrastive vision-language models. Despite achieving over 95% Species Accuracy, BioCLIP2 obtains substantially lower HCA (57.53% on Plant and 64.61% on Animal). BioCLIP exhibits a similar discrepancy. These findings suggest that the gap between fine-grained recognition and multi-level hierarchical recognition is not exclusive to generative MLLMs, but also exists in biodiversity-specific contrastive models.
Evaluation Protocol. We evaluate the official BioCLIP[1] and BioCLIP2[2] checkpoints using the same evaluation images, ordered four-candidate sets at each taxonomic level, ground-truth labels, and taxonomy mapping as in our main experiments. Following their native contrastive inference paradigm, we construct candidate descriptions using the prompt "a photo of a {category_name}." Image and text embeddings are independently L2-normalized, and candidates are ranked by cosine similarity at each taxonomic level. The resulting predictions are evaluated using the same hierarchical metrics as PhyRC.
Table. Evaluation of Contrastive Vision-Language Models on iNat21 (%).
| Model | Dataset | HCA | POR | S-POR | TOR | Leaf |
|---|---|---|---|---|---|---|
| BioCLIP | iNat-Plant | 21.92 | 68.36 | 62.71 | 50.78 | 89.30 |
| BioCLIP2 | iNat-Plant | 57.53 | 85.88 | 82.26 | 76.43 | 95.41 |
| BioCLIP | iNat-Animal | 42.84 | 78.83 | 71.09 | 61.75 | 88.96 |
| BioCLIP2 | iNat-Animal | 64.61 | 88.73 | 83.51 | 78.10 | 95.92 |
Analysis and Implications. The results reveal a substantial gap between Species Accuracy and hierarchical recognition performance. These results indicate that strong fine-grained species recognition does not necessarily translate into simultaneously correct predictions across taxonomic levels. Importantly, this phenomenon also occurs in biodiversity-specific contrastive vision-language models, suggesting that hierarchical prediction inconsistency is not specific to autoregressive generation architectures. We emphasize that this experiment investigates the generality of the observed phenomenon rather than comparing the effectiveness of different post-training methods. The results therefore extend our investigation of hierarchical recognition inconsistency across different vision-language architectures. We will incorporate these experiments and corresponding discussions into the revised manuscript.
[1] Stevens, Samuel, et al. "Bioclip: A vision foundation model for the tree of life." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.
[2] Gu, Jianyang, et al. "Bioclip 2: Emergent properties from scaling hierarchical contrastive learning." Advances in Neural Information Processing Systems 38 (2025).
(3/4)Author rebuttal
Q2: Cross-Dataset Evaluation on Visually Confusing Species
We thank the reviewer for this valuable suggestion. To assess whether PhyRC maintains taxonomic consistency when fine-grained visual differences are limited, we conducted a zero-shot cross-dataset transfer evaluation on CrypticBio[1], a benchmark featuring visually similar and difficult-to-distinguish species. Without any CrypticBio-specific training or model selection, PhyRC outperforms RFT by 2.53 pp in Leaf Accuracy and 1.13 pp in HCA, demonstrating improved fine-grained recognition and full-path correctness under challenging visual conditions.
Experimental Setup. We conduct a zero-shot cross-dataset transfer evaluation using the official CrypticBio test records and released candidate groups. We compare three Qwen3-VL-2B-Instruct model variants: the original Vanilla model and the RFT and PhyRC variants, both post-trained exclusively on the same iNaturalist-2021 one-shot training split. RFT and PhyRC are evaluated over three matched seeds. No CrypticBio data are used for training, fine-tuning, checkpoint selection, or hyperparameter tuning. To ensure deterministic evaluation, we freeze a common ordering of the released candidate sets; for Fungi, which are not covered by the original plant/animal prompt, we replace only the query noun with “organism.” We apply the hierarchical metrics defined in the main paper using a shared, paper-faithful implementation. LEAF accuracy is computed on the fixed cohort of 3,613 readable images with valid leaf labels and candidate membership, whereas HCA, POR, S-POR, and TOR are computed on the shared subset of 2,187 readable images with complete six-rank taxonomic annotations. All model variants use identical sample cohorts, ordered candidate sets, per-sample prompts, preprocessing, scoring rules, and metric implementations.
Table. Zero-shot cross-dataset transfer results on CrypticBio (%).
| Method | HCA | POR | S-POR | TOR | LEAF |
|---|---|---|---|---|---|
| Vanilla Qwen3-VL-2B-Instruct | 24.92 | 77.04 | 73.69 | 69.62 | 24.72 |
| RFT (3-seed mean ± std) | 25.19±0.86 | 76.61±0.25 | 73.85±0.14 | 69.75±0.09 | 25.46±0.35 |
| PhyRC (3-seed mean ± std) | 26.32±1.34 | 77.32±0.14 | 74.64±0.19 | 70.41±0.20 | 27.99±0.52 |
| Δ (PhyRC − RFT) | +1.13 | +0.71 | +0.79 | +0.66 | +2.53 |
The results show that PhyRC consistently outperforms both Vanilla and compute-matched RFT in LEAF accuracy and HCA. In particular, the improvements in both leaf-level recognition and complete-path accuracy indicate that PhyRC provides benefits beyond species-level classification alone. PhyRC also achieves consistent gains over RFT across POR, S-POR, and TOR, although the magnitude of improvement varies across metrics. These findings provide evidence of improved cross-dataset generalization to visually confusing species and stronger taxonomic consistency under challenging fine-grained recognition conditions.
We will incorporate the CrypticBio evaluation, including the experimental protocol, comparative results, and corresponding analysis, into the revised manuscript. This additional evaluation further validates the generalization capability of PhyRC beyond the iNaturalist-2021 benchmark.
[1] Manolache, Georgiana, Gerard Schouten, and Joaquin Vanschoren. "Crypticbio: A large multimodal dataset for visually confusing species." Advances in Neural Information Processing Systems 38 (2025).
(1/4)Author rebuttal
Response to Critical Weaknesses: Dataset Composition, Evaluation Scope, and Limitations
We thank the reviewer for the constructive comments. Our dataset audit confirms balanced species-level sampling, while additional evaluations of API-based MLLMs provide further evidence that hierarchical recognition challenges extend beyond the model families evaluated in our main experiments. Below, we clarify the scope of these findings and outline the limitations we will discuss in a dedicated section of the revised manuscript.
(1) Dataset composition and class distribution.
To address the reviewer's concern that the dataset composition and the number of images per species were not sufficiently clarified in the original manuscript, we conducted an additional audit of the actual post-training and evaluation manifests and will clarify these details in the revision. Specifically, the iNat-Plant and iNat-Animal evaluation sets contain 4,271 and 5,388 images, respectively, with exactly one image per species. Their taxonomic distributions are summarized below.
Importantly, the actual task-specific post-training dataset also contains one unique image per ground-truth species. The inat2021-mini dataset serves as the source pool, rather than representing the actual number of images used for post-training.
Consequently, conventional Head/Medium/Tail frequency-based analysis at the species level is not directly applicable to our actual training distribution. We therefore report macro-averaged accuracy and Task-Seen/Task-Unseen performance to provide complementary evaluations across species. Nevertheless, we acknowledge that the current balanced one-shot setting does not establish robustness under naturally long-tailed training distributions or account for variations in within-species visual diversity.
(2) Evaluation of multimodal large language models (MLLMs).
We acknowledge the reviewer's suggestion to extend the evaluation to proprietary multimodal models. Recent work by Tan et al. [1], The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition (CVPR 2026), has also reported taxonomic consistency difficulties in models such as GPT-4o.
Following this observation, we conducted additional evaluations using DeepSeek-V4.1-Flash and Kimi K2.6 through their official APIs on iNaturalist-2021. The results further confirm the presence of hierarchical recognition difficulties in multimodal large language models (MLLMs). Detailed protocols and results are provided in (4/4) Author Rebuttal.
(3) Limitations and Generalizability.
We will add a dedicated Limitations section addressing three aspects: (1) taxonomic coverage, currently focused on Plant and Animal; (2) architectural generalizability, as PhyRC primarily targets generative multimodal models; and (3) biodiversity scenarios, including naturally long-tailed distributions and within-species visual diversity. These directions will also inform future extensions of our work.
(4) Manuscript correction.
We thank the reviewer for pointing out this issue. We will add an explicit in-text cross-reference to the corresponding table on page 8 in the revised manuscript to improve clarity and readability.
[1] Tan, Yuwen, Yuan Qing, and Boqing Gong. "The llm bottleneck: Why open-source vision llms struggle with hierarchical visual recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.