acceptodds
Withdrawn from ICLR 2027

FishNet 2.0: A Comprehensive Multimodal Benchmark for Long-Tailed Fine-Grained Fish Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have demonstrated impressive cross-domain capabilities; however, their proficiency in specialized scientific domains such as marine biology remains underexplored. To systematically study this gap, we introduce FishNet2.0, a large-scale multimodal benchmark for long-tailed fine-grained fish understanding, comprising \newspe species with over 700,000 dense key-part annotations and more than 400,000 curated question-answer pairs. Beyond traditional recognition, FishNet2.0 integrates structured morphological descriptions and dense part-level localizations, enabling comprehensive evaluation across classification, detection, and grounded reasoning tasks. Extensive benchmarking of state-of-the-art MLLMs reveals severe degradation under real-world long-tailed distributions, with top-performing open-source models achieving below 40% accuracy. To diagnose the underlying causes, we propose a controlled evaluation framework that disentangles domain-specific knowledge from pure visual perception. Our analysis shows that performance failures stem primarily from insufficient domain knowledge rather than perceptual limitations. Importantly, we demonstrate that leveraging structured expert annotations, particularly explicit morphological descriptions and keypoint guidance, substantially improves zero-shot recognition performance. This establishes FishNet2.0 not only as a rigorous benchmark but also as a testbed for developing domain-aware multimodal reasoning strategies. Data and code will be released upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
MLLM, Ecology, Marine

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n2scOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FishNet 2.0, a large-scale multimodal benchmark for long-tailed fine-grained fish understanding. Building on FishNet 1.0, it expands a label-only dataset into a multi-annotated resource: it covers 99,556 images across 17,393 species, accompanied by 706,426 keypoint annotations, 119,399 bounding boxes, 17,393 expert-level species descriptions, and 418,623 question-answer pairs, supporting tasks including classification, detection, part localization, and visual question answering. To evaluate current multimodal large language models (MLLMs) in this specialized domain, the authors design a diagnostic framework that separately measures domain-specific knowledge and pure visual perception. They report that even the strongest open-source MLLMs achieve below 40% overall accuracy under the real long-tail distribution, and attribute the primary failure to insufficient domain knowledge rather than visual limitations. The paper further demonstrates that feeding species descriptions and keypoints into MLLMs as zero-shot contextual cues substantially improves recognition performance. In summary, the work's main contribution is the construction of a richly annotated benchmark and a diagnostic perspective for assessing MLLMs in a professional biological domain, rather than a new modeling method.

Critical strengths

1.The benchmark is large-scale and task-diverse, going beyond label-only datasets.

2.The motivation around long-tail fine-grained recognition in a specialized scientific domain is well-founded.

3.Separating domain knowledge from visual perception is a conceptually valuable direction.

Critical weaknesses

1.The majority of textual descriptions are GPT-4o-generated from heterogeneous web sources, with verification limited to 50 expert-checked entries (0.14%). No statistics are given on how many descriptions contain visually discriminative traits, and no ablation tests whether the reported gains come from genuine knowledge or text-distribution matching.

2.GPT-4o is outdated relative to the writing/release timeline, yet no rationale, comparison, or cross-check with stronger contemporary models is provided. Since GPT-4o also serves as the evaluation baseline, this creates a self-referential comparison.

3.The main diagnostic claim lacks rigorous causal support.​ The knowledge vs. perception split is not fully disentangled — text-only tasks may retain implicit visual priors, while vision-only tasks remain embedded in a domain-specific long-tail. Controlled counterfactuals (e.g., cross-species description swaps) are absent, making the attribution speculative.

4.Key claims lean on a substantial number of non-peer-reviewed preprints and technical reports. Critical dataset and baseline comparisons should rest on peer-reviewed sources wherever possible, and concurrent work should be clearly labeled.

5.The licensing chain for data aggregated from FishBase, iNaturalist, NOAA and others is unclear, nor whether GPT-4o-rewritten descriptions can be redistributed as a new annotation asset — a material concern at this scale.

6.No jointly trained baseline; modeling contribution is thin.​ The paper only applies RAG and LoRA on top of existing MLLMs, without an end-to-end model jointly trained on images, descriptions, and keypoints. The full ablation matrix separating descriptions, boxes, and keypoints is also missing, so the SFT gain cannot be causally attributed.

Questions for authors

1.What fraction of the 17,393 (or 35,133) descriptions contain visually discriminative morphological traits? Can you provide a random audit with expert discriminability ratings?

2.Why was GPT-4o chosen over stronger contemporary language models? Have you compared description quality across models?

Reviewer_ZNfROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce FishNet2.0, which is a large-scale multimodal benchmark for long-tailed fine-grained fish understanding. It comprises 17,393 species with over 700,000 dense key-part annotations and more than 400,000 curated question-answer pairs. FishNet2.0 integrates structured morphological descriptions and dense part-level localizations, enabling comprehensive evalu-ation across classiffcation, detection, and grounded reasoning tasks. Extensive benchmarking of state-of-the-art MLLMs reveals severe degradation under real-world long-tailed distributions. The authors also propose a controlled evaluation framework that disentangles domain-speciffc knowledge from pure visual perception to diagnose the causes.

Critical strengths
  1. Good annotation effort. The dataset curation requires good effort. The authors successfully pairing raw imagery with structured, domain-specific features like dense anatomical keypoints (eyes, fins, tail boundaries) and morphological descriptions.
  2. Good design of experiment for disentangling domain-speciffc knowledge.
Critical weaknesses
  1. Limited improvement over Fishnet. The original FishNet dataset already contained 94,532 images across 17,357 species. FishNet 2.0 adds 5,024 images (+5.3%) and 36 species (which the authors note on page 5 do not correspond to any known species in the taxonomy), along with outsourced bounding-box/keypoint annotations and GPT-4o-synthesized descriptions and QA pairs. While the keypoint annotation effort is substantial, augmenting an existing dataset and running off-the-shelf evaluations without proposing a new representation learning or long-tailed methodology may be not strong eough.
  2. Lack of Novelty in models and/or methods. The paper introduces no novel architectures, multimodal alignment techniques, or reasoning frameworks, relying entirely on standard benchmarking and basic LoRA fine-tuning.
  3. Overly relying on GPT-4o. The species descriptions and QA pairs were predominantly synthesized and evaluated by GPT-4o. Although the authors conducted a sample-based expert verification of 200 species, using closed-source, outdated models to generate the bulk of the textual ground truth introduces significant risks.
Questions for authors
  1. What are the reasons of using GPT-4o, since it is already sunset?
  2. How are 1-image Tail species split into train and test?
  3. The diagnostic experiments conclude that zero-shot MLLMs fail largely because they lack memorized domain knowledge, so why not having a RAG as baseline?
Reviewer_RoqXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present FishNet 2.0, a dataset that provides nearly 100,000 images of fish across a long-tailed species distribution. These images are paired with species labels, part locations, text descriptions, and question-answer pairs. The authors design this dataset for several downstream uses: fine-grained classification, object detection, keypoint localization, text-based question answering, and visual question answering. The authors also provide comprehensive tests of existing MLLMs on FishNet 2.0, helping to explore the ways in which current approaches underperform in fine-grained VQA scenarios.

Critical strengths
  1. The data scale presented in this work is impressive. Few datasets provide well-annotated fine-grained fish morphology, and fewer at large scale. At nearly 100,000 images and spanning several tasks, the presented dataset seems to be an effective testing ground for a range of AI methodologies, including MLLMs.
  2. Paired species descriptions with images are not necessarily novel (see BioCAP, ICLR 2026), but the methodology for doing so is sound and datasets like this are still beneficial
  3. Comprehensive experiments using FishNet 2.0 demonstrate meaningful results about MLLM performance in fine-grained, domain science-specific scenarios.
Critical weaknesses
  1. The methodology for annotation and expert verification is not clear. No details are given for how species descriptions are evaluated by experts or users, making it difficult to gauge the accuracy of the provided scores. a. Additionally, it is not clear if expert evaluation covers all species descriptions, or just a small subset of species descriptions. If expert evaluations do not cover all descriptions, then this presents a possible weakness of the work
  2. While GPT-4o has proved to have high agreement with humans in other datasets (ThinkSLM), it has not been clearly established by the authors that this is the case in species descriptions. This could also be addressed by further detailing the expert and user evaluation protocols.
  3. The authors fail to compare this work to extremely relevant existing work (FishVista). While this dataset may still improve on existing work, this is a needed comparison.
Questions for authors
  1. What is the exact protocol and coverage for expert evaluation of descriptions? The authors state that annotators were shown images of species alongside descriptions and asked to rate how helpful the description was. Did these evaluations cover all species in the dataset?
  2. Are the experts being asked to rate helpfulness in relation to a specific task like species classification?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.