- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FishNet 2.0, a large-scale multimodal benchmark for long-tailed fine-grained fish understanding. Building on FishNet 1.0, it expands a label-only dataset into a multi-annotated resource: it covers 99,556 images across 17,393 species, accompanied by 706,426 keypoint annotations, 119,399 bounding boxes, 17,393 expert-level species descriptions, and 418,623 question-answer pairs, supporting tasks including classification, detection, part localization, and visual question answering. To evaluate current multimodal large language models (MLLMs) in this specialized domain, the authors design a diagnostic framework that separately measures domain-specific knowledge and pure visual perception. They report that even the strongest open-source MLLMs achieve below 40% overall accuracy under the real long-tail distribution, and attribute the primary failure to insufficient domain knowledge rather than visual limitations. The paper further demonstrates that feeding species descriptions and keypoints into MLLMs as zero-shot contextual cues substantially improves recognition performance. In summary, the work's main contribution is the construction of a richly annotated benchmark and a diagnostic perspective for assessing MLLMs in a professional biological domain, rather than a new modeling method.
Critical strengths
1.The benchmark is large-scale and task-diverse, going beyond label-only datasets.
2.The motivation around long-tail fine-grained recognition in a specialized scientific domain is well-founded.
3.Separating domain knowledge from visual perception is a conceptually valuable direction.
Critical weaknesses
1.The majority of textual descriptions are GPT-4o-generated from heterogeneous web sources, with verification limited to 50 expert-checked entries (0.14%). No statistics are given on how many descriptions contain visually discriminative traits, and no ablation tests whether the reported gains come from genuine knowledge or text-distribution matching.
2.GPT-4o is outdated relative to the writing/release timeline, yet no rationale, comparison, or cross-check with stronger contemporary models is provided. Since GPT-4o also serves as the evaluation baseline, this creates a self-referential comparison.
3.The main diagnostic claim lacks rigorous causal support. The knowledge vs. perception split is not fully disentangled — text-only tasks may retain implicit visual priors, while vision-only tasks remain embedded in a domain-specific long-tail. Controlled counterfactuals (e.g., cross-species description swaps) are absent, making the attribution speculative.
4.Key claims lean on a substantial number of non-peer-reviewed preprints and technical reports. Critical dataset and baseline comparisons should rest on peer-reviewed sources wherever possible, and concurrent work should be clearly labeled.
5.The licensing chain for data aggregated from FishBase, iNaturalist, NOAA and others is unclear, nor whether GPT-4o-rewritten descriptions can be redistributed as a new annotation asset — a material concern at this scale.
6.No jointly trained baseline; modeling contribution is thin. The paper only applies RAG and LoRA on top of existing MLLMs, without an end-to-end model jointly trained on images, descriptions, and keypoints. The full ablation matrix separating descriptions, boxes, and keypoints is also missing, so the SFT gain cannot be causally attributed.
Questions for authors
1.What fraction of the 17,393 (or 35,133) descriptions contain visually discriminative morphological traits? Can you provide a random audit with expert discriminability ratings?
2.Why was GPT-4o chosen over stronger contemporary language models? Have you compared description quality across models?