- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BioBench, a benchmark that tests how well multimodal LLMs understand species. It contains 5,919 images of 3,611 species and 8,821 multiple-choice questions, grouped into five tasks that range from recognizing body features, taxonomic rank and species to answering biological and human-related knowledge questions. The data mainly comes from BioCLIP data matched against the IUCN Red List, and most descriptions and questions are written by Qwen models and GPT-4o and then filtered. The authors also build a 160K instruction dataset called BioInstruct and fine-tune Qwen2.5-VL-3B on it. The authors evaluate 25 existing models. According to the paper the best of them is Qwen2.5-VL-7B with an average of 59.48%. Its accuracy falls from 62.68 on easy species identification to 34.85 on hard, and R1-style reasoning models do not help much. Fine-tuning on BioInstruct raises the 3B model from 57.02 to 61.20.
Critical strengths
- The species coverage is broad. The benchmark spans 3,611 species of animals, plants and fungi, which goes well beyond earlier biology benchmarks such as VLM4Bio that only cover fishes, birds and butterflies.
- The difficulty design is thoughtful. Taxonomic classification is split by level and species identification is graded by taxonomic distance with BioCLIP-ranked distractors, so the benchmark shows where models break down instead of giving a single number. The drop from 62.68 on easy species identification to 34.85 on hard is a good example of this.
- Masking the species name and removing questions that can be answered without the image deals with a known weakness of multimodal benchmarks, where many questions can be solved from text alone. The training setup and the prompts used for generation are also described in enough detail to follow.
Critical weaknesses
-
Unreliable Leaderboard Metrics: The reported leaderboard scores appear statistically inconsistent with the dataset size. For example, Figure 2 lists 1,779 questions for hard species identification and 145 questions for the discriminative MAR split. However, for most models, the reported percentages cannot be back-computed to any whole number of correct answers based on these denominators, strongly suggesting that different models were scored on different subsets of questions. Furthermore, the InternVL models exhibit oddly weak and clustered performance (e.g., the 38B InternVL3 scores 40.89%, falling behind the 7B LLaVA-v1.5 at 42.72%), which raises concerns about whether appropriate inference templates were used. This scoring methodology must be audited and standardized before the rankings can be trusted.
-
Opaque Question Construction and Filtering: The pipeline for building and filtering questions lacks necessary detail. For the MAR task, the paper simply states that the base setting has "coarse differences" and the discriminative setting has "fine-grained ones," without explaining how the correct and distractor descriptions are systematically generated. Moreover, one filtering step relies on several "unnamed MLLMs"; if these include models later evaluated in the paper, the filtering process inherently biases their scores. Finally, the manual verification step is described in a single sentence without reporting the sample size or error rate, making it difficult to verify the claim that the question set is highly reliable.
-
Contradictions Between Claims and Data: Several central claims contradict the paper's own tables. The text asserts that Qwen2.5-VL-7B is the best model at 59.48%, yet Appendix Table 5 shows Qwen2.5-VL-72B achieving 66.55%. The abstract claims BioQwenVL surpasses "larger frontier models", but no closed-source models are tested, and its 61.20% score falls below both the 32B (64.62%) and 72B Qwen models. Furthermore, the full BioQwenVL model averages 59.54% in Table 4 but 61.20% in Table 3. Lastly, it is unclear whether BioInstruct and BioBench share species or images, making it impossible to determine how much of the 4.18% performance gain is true generalization.
-
Knowledge Tasks Conflated with Visual Recognition: The knowledge tasks likely measure species recognition rather than reasoning capability. A question is retained only if text-only LLMs answer it correctly when provided the true species name, and it is removed if it can be answered without the image once the name is masked. Consequently, the surviving questions are fundamentally reliant on the model recognizing the species visually. As the authors themselves note, knowledge reasoning depends heavily on accurate species identification. Without providing ablation results where the model is explicitly given the true species name, a low score could simply indicate a visual misclassification rather than a lack of biological knowledge. Therefore, the conclusion that reasoning models struggle to apply biological knowledge is insufficiently supported.
Questions for authors
-
Do BioBench and BioInstruct share species or images? Reporting the performance gain separately for species seen and unseen during training would help clarify the true generalization capability.
-
How exactly are the MAR options generated, which specific MLLMs were used for filtering, and what were the sample size and error rate for the manual verification step?
-
How do the evaluated models score on the knowledge tasks when they are explicitly provided with the true species name in the prompt?