acceptodds
Under review as a conference paper at ICLR 2027

ShellfishNet: A Domain-Specific Benchmark for Visual Recognition of Marine Molluscs

Abstract

Reliable visual recognition of shellfish is important for ecological monitoring and aquaculture, but real-world imagery exhibits substantial variation in acquisition conditions, appearance, and posture. Existing marine vision resources cover diverse taxa and imaging settings, but fine-grained mollusc recognition remains underexplored in a group-disjoint benchmark that jointly evaluates class prediction, representation-space retrieval, and condition sensitivity. To address this gap, we introduce ShellfishNet, a provenance-aware, group-disjoint benchmark of 40,130 images from 100 marine mollusc classes, organized into 30,879 acquisition or observation groups. Rather than treating recognition as a single clean-accuracy leaderboard, ShellfishNet evaluates model configurations across classification, image-to-image retrieval, taxonomic error structure, and controlled condition sensitivity. We benchmark representative convolutional, transformer-based, hybrid and efficient, self-supervised, and vision-language models on classification and retrieval. Separately, we evaluate open image-captioning and vision-language models on the captioning subset and assess selected fixed checkpoints under controlled generic and marine-inspired synthetic transformations. Across the benchmark, image-level and class-aware classification largely agree, whereas retrieval provides complementary model rankings and taxonomy projection reveals that a non-negligible subset of exact-class errors remains correct at higher taxonomic ranks. Selected fixed checkpoints with similar clean accuracy can also exhibit distinct sensitivity profiles under controlled transformations. These results show that model comparison on shellfish imagery should account for both task and image condition, rather than rely on a single clean classification score. The dataset and evaluation protocols will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
marine computer vision, fine-grained visual recognition, benchmark datasets, mollusc recognition, image retrieval, image captioning, vision-language models, synthetic transformation sensitivity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_j5epOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a shellfish benchmark for visual recognition of marine molluscs. This benchmark is large and comprehensive, including both visual images and a textual corpus. Based on that, the authors benchmark this dataset with various tasks, including closed-set recognition and image-to-image retrieval. Besides, they also examine the taxonomic error structure and controlled condition sensitivity. To sum up, this benchmark is practical and will be valuable for potential multimodal research in field in the future.

Critical strengths
  1. This dataset may be the largest publicly available curated dataset of molluscs to date. The authors have integrated data from multiple existing databases and other sources, carefully organized and cleaned the collection, and provided corresponding textual descriptions through captioning. These paired image–text data would facilitate future multimodal research on molluscs, making the dataset highly valuable for practical use.
  2. The authors conducted extensive benchmarking on this dataset, evaluating nearly 80 backbones to assess its difficulty. These experiments characterize the range of performance achieved by existing models and provide the first systematic assessment of how current state-of-the-art models perform in this specialized domain. This extensive evaluation is also one of the paper’s concrete contributions.
Critical weaknesses

None.

Questions for authors
  1. What does observation group mean? I could not find a definition in the paper. If I have overlooked it, please point me to the relevant explanation and clarify the term’s meaning and significance.
  2. I am interested in the details of the dataset deduplication procedure. Although the authors provide a general overview in the appendix, the specific steps are not described in sufficient detail.
  3. I am curious about the dataset’s level of fine-grained detail and the similarity between classes. The authors mention that some classes may be highly similar but do not provide supporting evidence. If possible, could the authors provide a confusion matrix to illustrate this?
4 comments
AuthorsOpenReview ↗

Response to Reviewer j5ep (1/3)

We thank the reviewer for the detailed feedback and for recognizing the value of the dataset and benchmark. We clarify observation groups, detail the deduplication procedure, and present species-level confusion evidence below. Additional analyses are provided in Appendix G.1–G.6 of the revised manuscript.

Q1. What does observation group mean?

I could not find a definition in the paper. If I have overlooked it, please point me to the relevant explanation and clarify the term’s meaning and significance.

An acquisition or observation group links images from the same capture event or source record. For example, two photographs attached to one observation belong to the same group. Section 3.1 (pp. 4–5) defines this term and explains the partitioning rule; Appendix A.2 (pp. 23–24) describes the released grouping metadata.

We construct the group identifier from the first available field in this order: acquisition-session identifier, observation identifier, specimen identifier, and source-platform plus source-record identifier. The released metadata stores this assignment as group_id.

Each group is assigned wholly to training, validation, or test. This keeps related views together and reduces leakage from shared subjects or capture conditions. Deduplication removes image copies; grouping keeps the remaining related photographs in one partition. ShellfishNet contains 40,130 images in 30,879 groups, with at most two images per group. No group crosses partitions. Figure 2(d) shows the group-size distribution, and Table 4 (p. 24) reports the partition audit.

AuthorsOpenReview ↗

Response to Reviewer j5ep (2/3)

Q2. I am interested in the details of the dataset deduplication procedure.

Although the authors provide a general overview in the appendix, the specific steps are not described in sufficient detail.

The procedure combines exact-image checks, transformed-image candidate retrieval, local verification, and conservative exclusion before partitioning. We give the operational steps below, supplementing the audit totals in Appendix A.2 and Table 4.

  1. Detect exact copies. We detect duplicate images by comparing file-level hashes and decoded-pixel hashes. The latter include image shape and RGB pixels after EXIF-orientation correction, detecting identical pixels stored in different files.

  2. Retrieve possible transformed copies. We compute 64-bit perceptual hashes (pHash) and difference hashes (dHash). Views include the full image and an 82% central crop, each rotated by 0°, 90°, 180°, and 270°. Candidate retrieval uses four 16-bit pHash segments, requiring at least two shared-segment bucket occurrences; buckets exceeding 500 images are skipped. Global similarity uses HSV histograms and resized grayscale images. Hash distances are minima across the image views. A permissive filter retains exact matches or pairs satisfying any of four conditions:

    • pHash distance ≤14;
    • pHash ≤20 with global similarity ≥0.82;
    • dHash ≤18 with global similarity ≥0.86;
    • global similarity ≥0.965.

    These rules select candidates for verification.

  3. Verify local correspondence. We extract up to 3,000 ORB features per image and apply a 0.75 nearest-neighbor ratio test. RANSAC estimates a homography with a 4-pixel reprojection threshold. Verification images have a maximum long edge of 1,200 pixels. We measure inlier count, inlier ratio, spatial coverage, and photometric normalized cross-correlation (NCC) after alignment. Exact file or decoded-pixel matches are accepted directly. Other pairs must satisfy every threshold in the applicable column below.

Final duplicate criterionSame scientific nameDifferent scientific names
Minimum pHash Hamming distance≤6≤2
Minimum dHash Hamming distance≤8≤2
RANSAC inliers≥45≥45
Inlier ratio≥0.55≥0.80
Spatial coverage≥0.20≥0.30
Aligned photometric NCC≥0.90≥0.85
  1. Separate decisions and select representatives. Pairs are classified as strict duplicates, unresolved candidates for review, or rejected duplicate hypotheses. Strict duplicate pairs form clusters, with one representative retained per cluster. Representative selection considers license eligibility, rights verification, original-source status, resolution, sharpness, watermarks, and provenance completeness. Unresolved candidate images are conservatively excluded from release selection. Pairs whose duplicate hypothesis is rejected retain both images.

  2. Audit the release and partition by group. After deduplication, selection applies the quality, licensing, class-size, and group constraints. We verify that excluded records are absent from both the selected and released collections. We then check the released record correspondence and group-disjoint partitions.

The audit covers 57,171 candidate images. Candidate retrieval produced 2,106,464 pairs, of which 194,024 underwent local verification. The final exclusions and release checks are:

Audit itemResult
Strict duplicate clusters216 clusters containing 442 images
Nonrepresentative cluster members removed226 images
Additional unresolved candidate paths excluded871 paths
Unique exclusions absent from selection and release1,097 / 1,097
Selected-to-released record correspondence40,130 / 40,130

The 871 additional exclusions come from 878 paths in 754 unresolved pairs, after removing seven paths already counted among the strict removals. The record-correspondence check uses stored checksums and source-identity fields. These checks connect the deduplication decisions to the released dataset and confirm zero observation-group overlap across partitions.

AuthorsOpenReview ↗

Response to Reviewer j5ep (3/3)

Q3. I am curious about the dataset’s level of fine-grained detail and the similarity between classes.

The authors mention that some classes may be highly similar but do not provide supporting evidence. If possible, could the authors provide a confusion matrix to illustrate this?

ShellfishNet tests species identification from ordinary photographs. The confusion analysis identifies retained species that remain difficult to distinguish across models and seeds. We analyzed full 100-class test confusion matrices for MaxViT-Tiny and DINOv2 ViT-B/14. We normalized each row, then averaged the matrices across seeds 41, 42, and 43 on the same 3,958 test images. Rows denote true species and columns denote predictions.

To examine specific confusions, we pooled the six validation confusion matrices from these two models and ranked species pairs by symmetric row-normalized confusion. We selected the ten highest-ranked pairs. We then measured their confusion on the held-out test set. All counts come from full 100-class predictions. For seed ss, the symmetric confusion rate is

Hs(i,j)=12[Cs(i,j)ni+Cs(j,i)nj],H_s(i,j)=\frac{1}{2}\left[\frac{C_s(i,j)}{n_i}+\frac{C_s(j,i)}{n_j}\right],

where Cs(i,j)C_s(i,j) counts true class-ii images predicted as jj, and nin_i is the class-ii test count. Larger values indicate greater confusion between the two species. The table reports three-seed mean HsH_s in percent:

Retained species pairTest images (nin_i / njn_j)MaxViT-Tiny (%)DINOv2 (%)
Busycon carica / Sinistrofulgur sinistrum47 / 4214.9416.22
Spisula solidissima / Tivela stultorum47 / 4513.309.24
Crassostrea virginica / Ostrea edulis49 / 497.487.82

For Busycon carica and Sinistrofulgur sinistrum, errors occur in both directions across all 23 original models and three seeds (69 runs). This repeated confusion provides direct evidence of species-recognition difficulty within the selected inventory. Appendix G.1 reports all ten validation-selected pairs in Table 25 (p. 52) and their individual-seed results in Figure 21 (p. 53). Figure 22 (p. 53) adds directional confusion matrices for the seven species in pairs 1–4. Rows give true species and columns give predictions; the “Other 93” column retains errors to the remaining classes. Each row sums to 100% before rounding. These named pairs make the recurring identification difficulties explicit.

Dear Reviewer j5ep,

We hope you are well. Thank you again for your careful review and helpful questions.

Our responses explain the observation groups and deduplication procedure, and the revised appendix provides additional evidence of interspecies confusion. When convenient, we would welcome your feedback on whether these explanations address your questions. If anything remains unclear, please let us know; we would be happy to clarify.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_BY1COpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces ShellfishNet, a 40,130-image, 100-class marine mollusc benchmark drawn from public biodiversity repositories, with per-image provenance and group-disjoint splits (32,143/4,029/3,958 images; 30,879 groups). It benchmarks 23 models over three seeds (69 runs) on classification and retrieval, adds a single-seed 80-backbone scan, a 512-image captioning study with blinded human scoring, and fixed-checkpoint synthetic-sensitivity tests. It concludes that model comparison should account for task and image condition rather than a single clean-accuracy score.

Critical strengths
  1. The dataset is well documented. Per-image sources, licenses, duplicate screening, release records, and group-disjoint splits make its provenance and construction easy to audit.

  2. The evaluation covers 23 models over three seeds on classification and retrieval. The captioning study adds blinded human ratings and agreement analysis, giving that part of the benchmark a useful check beyond automated scores.

Critical weaknesses
  1. The class selection does not establish the claimed fine-grained challenge. Appendix A.1 excludes taxa with severe visual overlap, and Table 2 retains one species per genus. This removes close within-genus comparisons, yet the paper provides no confusion analysis showing which retained pairs remain difficult. Per-class Recall/F1 alone does not establish the subtle distinctions that motivate the benchmark.

  2. The model ranking may depend heavily on the training recipe. Two recipes tuned on MobileNetV3 and ViT-B/16 are assigned by family without per-model tuning; the top six models span only 2.72 percentage points. SwinV2-Tiny and RepViT-M1.0 are unexplained outliers. The separate HERBS/Swin-L run improves on the leaderboard by 7.27 points but changes backbone, resolution, and training setup. Targeted recipe checks are needed to establish how much of the ranking is stable.

  3. The taxonomic and label checks provide limited additional validation. With 100 species from 100 distinct genera, genus projection simply relabels predictions; family/order gains follow from merging classes. These results do not establish fine-grained discrimination. Labels were checked by one reviewer, without independent second-expert agreement, so their reliability is also not quantified.

  4. The condition analyses offer limited evidence of generalization. Each synthetic-sensitivity panel uses five fixed seed-41 checkpoints, and the marine transforms are empirical RGB perturbations. Per-class test sets contain only 13–50 images, without sampling uncertainty for class or stratum comparisons. The external official-test subset has just 25 images from seven taxa. These results are too narrow to establish robustness under real acquisition changes.

Questions for authors
  1. Which retained class pairs are hardest? A confusion analysis or expert difficulty ratings would help establish the fine-grained challenge in the selected inventory.

  2. How stable are the top-six ordering and the SwinV2-Tiny/RepViT-M1.0 outliers under alternative training recipes?

  3. Beyond the 25-image external test, are there larger source- or geography-disjoint evaluations, or results on real degraded images, that reproduce the reported failures?

17 comments
AuthorsOpenReview ↗

Response to Reviewer BY1C (1/3)

We thank the reviewer for the detailed feedback. We address W1–W4 and Q1–Q3 below, with supporting analyses in Appendix G.1–G.6 of the revised manuscript.

W1. The class selection does not establish the claimed fine-grained challenge.

The added confusion analysis identifies recurring errors between retained species. The inventory targets identification from ordinary photographs and contains one species per genus. Screening excluded severe visual overlap and taxa requiring anatomy, microscopy, genetics, or geographic context (Appendix A.1, Table 2). Its scope is cross-genus species recognition; it does not test within-genus discrimination.

We selected ten pairs using validation predictions from MaxViT-Tiny and DINOv2 ViT-B/14 across seeds 41, 42, and 43. We pooled the six validation confusion matrices and ranked pairs by symmetric row-normalized confusion, then evaluated held-out test confusion. For seed ss, symmetric confusion is Hs(i,j)=12[Cs(i,j)ni+Cs(j,i)nj]H_s(i,j)=\frac{1}{2}\left[\frac{C_s(i,j)}{n_i}+\frac{C_s(j,i)}{n_j}\right]. Here, Cs(i,j)C_s(i,j) counts class-ii images predicted as jj, and nin_i is the class-ii test count. Larger values indicate greater confusion between the two species. Counts come from full 100-class predictions. Examples below report three-seed mean HsH_s in percent:

Retained species pairTest imagesMaxViT-Tiny (%)DINOv2 (%)
Busycon carica / Sinistrofulgur sinistrum47 / 4214.9416.22
Spisula solidissima / Tivela stultorum47 / 4513.309.24
Crassostrea virginica / Ostrea edulis49 / 497.487.82

For the first pair, both error directions occur in all 69 original model/seed runs. Appendix G.1 reports all ten pairs in Table 25 and their seed-level results in Figure 21; Figure 22 shows directional confusion for the seven species in pairs 1–4. This directly supplements per-class scores with evidence of repeated interspecies confusion.

W2. The model ranking may depend heavily on the training recipe.

The exact top-six ordering changes under the additional recipe checks. We tested the original top six and both outliers at {1/3,1,3}\{1/3,1,3\} times each model's original learning rate (LR). Each run used 100 epochs, 224-pixel inputs, batch size 48, and the official splits. Seed-41 validation Macro-F1 selected the LR; the selected recipe was then run with seeds 42 and 43. Checkpoint selection also used validation Macro-F1, giving 40 runs.

Top-1 accuracy is reported as mean ± sample SD across three seeds:

ModelSelected LR factorSubmitted protocol (%)Additional protocol (%)
DINOv2 ViT-B/141/382.71 ± 0.9487.56 ± 0.28
MaxViT-Tiny384.39 ± 0.2286.01 ± 0.37
RegNetY-8GF183.33 ± 0.4783.35 ± 0.57
ConvNeXt-Tiny382.10 ± 0.4282.87 ± 0.09
EfficientNetV2-S182.35 ± 0.2982.33 ± 0.37
Swin-Tiny181.67 ± 0.4381.62 ± 0.35
RepViT-M1.01/351.37 ± 2.6361.01 ± 3.59
SwinV2-Tiny349.08 ± 6.6453.19 ± 2.02

Three of the 15 top-six pairwise orderings reverse (Kendall τ=0.60\tau=0.60). DINOv2 replaces MaxViT at the top. Within the eight tested models, the original top six remain ahead of both outliers. The table distinguishes submitted and additional protocols (also Appendix G.2, Table 27); LR effects below use matched controls trained in the same environment.

At seed 41, lowering RepViT's LR to one-third raises accuracy from 53.39% to 58.08%. Tripling SwinV2's LR raises accuracy from 38.23% to 54.45%. The selected SwinV2 runs peak in validation Macro-F1 at epochs 7/14/7, followed by rising training loss and declining validation performance. Using one-third of the original LR delays the seed-41 peak to epoch 50 and reduces the subsequent decline. The rising training loss points to optimization instability rather than simply too few epochs. The LR response shows that its severity depends on the tested recipe. Appendix G.2 provides the full validation LR grid (Table 26) and matched LR comparisons (Figure 23). Figure 24 shows unsmoothed seed-41 curves; Table 28 gives SwinV2 peak and final values across the tested runs.

The leaderboard reports performance under specified training configurations. The 80-backbone, single-seed, 30-epoch scan provides broader descriptive coverage (Appendix B.4). HERBS/Swin-L at 384 pixels is a separate configuration (Appendix B.3, Table 10); its 7.27-point gap combines method, backbone, resolution, and training changes.

AuthorsOpenReview ↗

Response to Reviewer BY1C (2/3)

W3. The taxonomic and label checks provide limited additional validation.

Genus projection is redundant in this inventory, and family/order accuracy gains arise from merging labels. The taxonomic analysis describes where errors occur. We added a prediction-label permutation baseline preserving predicted-class frequencies. For MaxViT, 21.69% of errors stay within the same family, versus 1.96% under this baseline, averaged across three seeds. This quantifies taxonomic concentration beyond that expected from class frequencies. Appendix G.3, Table 29 extends this comparison to every original model at both family and order levels.

The original curation used one reviewer familiar with shellfish taxonomy (Appendix A.2). Accepted-name checks and image-level provenance provide the audit trail for this curation. We have now added a label audit by a second expert on 100 images randomly sampled from the full dataset. The initial review flagged nine cases for closer taxonomic examination. Targeted re-examination confirmed the existing species labels in all nine cases. No species-label errors were identified in the 100-image sample after this two-stage audit. We will publish the annotation protocol on Hugging Face and use public Issues to collect and respond to label feedback. We will review reported errors and document confirmed label corrections in subsequent dataset releases.

W4. The condition analyses offer limited evidence of generalization.

We expanded both sensitivity panels to three seeds and added 10,000-resample, within-class observation-group bootstrap intervals. The panels cover eight distinct models in total. Test-sampling intervals are conditional on the evaluated checkpoints; training-seed SD is reported separately. Per-class and scene-stratified intervals quantify sampling uncertainty for the existing 13–50 test images per class. Appendix G.4 provides the per-class interval-width summary (Table 30), scene intervals (Table 31), and three-seed sensitivity plots (Figure 25).

For example, DINOv2's Jerlov-II accuracy drop is 10.88 percentage points (95% group interval: 10.07–11.69; seed SD: 0.55). The marine transforms measure sensitivity to empirical RGB perturbations; they are not depth-resolved underwater simulations.

We also constructed a split with no cross-partition observation groups or normalized photographer strings. The photographer-string-disjoint and group-only control designs match per-class image budgets, each covering 100 classes and 4,023 test images. We trained three models with the original fixed recipes and two seeds per design. Macro-F1 is mean ± sample SD:

ModelGroup-only control (%)Photographer-string-disjoint (%)
ResNet-5079.70 ± 0.0376.46 ± 0.84
MaxViT-Tiny85.99 ± 1.1182.96 ± 0.04
DINOv2 ViT-B/1483.13 ± 0.6881.47 ± 0.81

This extends acquisition-related evaluation across all 100 classes; Q3 adds a source-platform holdout on real photographs. Photographer strings serve as identity proxies, and the two designs also differ in test images, sources, and scenes. Their scores describe the combined acquisition differences. Appendix G.5 details the split and its control; Table 32 and Figure 26 report their separate sampling intervals.

AuthorsOpenReview ↗

Response to Reviewer BY1C (3/3)

Q1. Which retained class pairs are hardest?

Among the validation-selected pairs, Busycon carica / Sinistrofulgur sinistrum and Spisula solidissima / Tivela stultorum remain difficult on test. See W1 for their confusion rates and Appendix G.1, Figures 21–22 for seed-level results and error directions.

Q2. How stable are the top-six ordering and the SwinV2-Tiny/RepViT-M1.0 outliers under alternative training recipes?

The original top six remain ahead of both outliers among the eight tested models, while their exact ordering changes. RepViT improves with lower LR. SwinV2's early validation peaks and subsequent training-loss increases identify optimization instability under the tested recipes (W2; Appendix G.2, Figures 23–24).

Q3. Beyond the 25-image external test, are there larger source- or geography-disjoint evaluations, or results on real degraded images, that reproduce the reported failures?

Yes. We added a source-platform holdout of 1,033 photographs from 86 species and 1,016 observation groups. These comprise all non-iNaturalist images: GBIF (940), Wikimedia Commons (74), iDigBio (14), and EOL (5). Training uses 8,264 iNaturalist images from all 100 classes; 1,033 separate iNaturalist images select checkpoints by validation Macro-F1. All partitions are observation-group-disjoint. We also retain an independent iNaturalist test with 3,281 images from the same 86 species, outside the nominal 8:1:1 allocation. Appendix G.6 details the partitions (Table 33) and duplicate screening. No training/validation image matched the held-out platform images by exact hash or perceptual hash within Hamming distance 4.

ResNet-50 and DINOv2 each use the original fixed recipe for 100 epochs with seeds 41 and 42. Each selected checkpoint evaluates both test domains using full 100-class predictions. Both metrics average over the shared 86 true classes. Values are percentages: mean ± seed sample SD [95% sampling interval]. Intervals use 10,000 within-class observation-group bootstrap resamples of the two-seed mean. Appendix G.6, Table 34 and Figure 27 report the two-domain comparison.

ModelTest domainMacro-F1 (%)Balanced accuracy (%)
ResNet-50iNaturalist63.88 ± 0.28 [62.24, 65.31]63.23 ± 0.39 [61.68, 64.84]
ResNet-50Held-out platforms39.96 ± 2.56 [37.24, 41.96]43.61 ± 2.27 [41.07, 46.14]
DINOv2 ViT-B/14iNaturalist77.73 ± 0.73 [76.25, 79.02]76.95 ± 0.60 [75.53, 78.38]
DINOv2 ViT-B/14Held-out platforms54.83 ± 1.66 [52.22, 57.41]60.66 ± 1.52 [58.13, 63.10]

Performance declines on held-out platforms for both architectures and both seeds. The pattern persists within controlled and intertidal scenes and among the 44 species with at least ten held-out groups. Excluding seven images with overlapping photographer strings changes held-out Macro-F1 by only +0.09/+0.06 points for ResNet-50/DINOv2. Appendix G.6, Table 35 and Figure 28 cover all predefined support, scene, and photographer-overlap checks. Table 36 adds results for each recorded source platform.

This extends acquisition-transfer evidence from the 25-image external diagnostic to a larger source-platform holdout on real photographs within ShellfishNet. It measures transfer across recorded platforms and their image mixtures; geography-disjoint and real-underwater degradation tests remain outside this evaluation.

ShellfishNet provides an auditable domain benchmark with direct confusion evidence, measured training sensitivity, and acquisition-transfer tests on real photographs.

In light of these additional results and clarifications, we would be grateful if the reviewer could reconsider their assessment.

Reviewer_yATmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a domain-specific benchmark named ShellfishNet for fine-grained visual recognition of marine molluscs. The dataset contains 40,130 images (100 species), organized into 30,879 acquisition or observation groups with provenance-aware, group-disjoint training, validation, and test splits. Its evaluation metrics have expanded from merely classification accuracy to include classification, image-to-image retrieval, classification error analysis, image captions, as well as sensitivity to general and marine transformations. The authors validate the benchmark across a diverse range of visual and multimodal models under different imaging conditions, providing a new evaluation resource for fine-grained mollusc recognition.

Critical strengths

(1) ShellfishNet, which contains 40,130 images from 100 species of molluscs, retains the image-level source, license information, taxonomic identity, and collection or observation group, providing new reusable standards.

(2) The group-disjoint splits and duplicate-removal audit ensure that the indicator results are traceable.

(3) The authers broadly evaluates classification, retrieval, captioning, taxonomic error structure, calibration, efficiency, and sensitivity to both generic and marine-inspired image changes,  supporting several practically relevant dimensions of marine visual recognition.

Critical weaknesses

(1) The paper states that class labels were checked by one reviewer familiar with shellfish taxonomy, but no independent expert verification or inter-annotator agreement is reported. However all classification, retrieval, and taxonomic-error results depend on the correctness of these labels. Even a modest rate of systematic species-level errors could materially affect the benchmark rankings and the interpretation of “taxonomically close” mistakes.

(2) Targeted synthetic sensitivity analyses are limited to independently applied synthetic RGB transformations. Jerlov transformation simulation is far from a depth-resolved physical simulation. It therefore remains unclear whether the reported sensitivity patterns generalize to real underwater images containing coupled effects such as attenuation, backscatter, nonuniform illumination, occlusion, and background clutter.

Questions for authors

(1) What qualifications and taxonomic resources were used by the reviewer who checked the species labels? How many source labels were corrected, rejected, or marked as ambiguous during curation?

(2) Appendix A.2 states that photographer identity, geographic location, and source platform are not used as partition keys, and that cross-platform identity resolution is limited by the available metadata. How much overlap in photographers, locations, time periods, and source platforms remains between the training, validation, and test sets?

(3) The two optimizer profiles used for the 23-model benchmark were selected using MobileNetV3-Large and ViT-B/16 as anchors, while the 80-backbone scan uses a single training recipe for 30 epochs. How did the authros determine that these settings provide adequate optimization for the remaining architectures? Could the low or unstable performance of models such as SwinV2-Tiny and RepViT-M1.0 be caused by unsuitable training settings or incomplete convergence?

(4) How were the 512 captioning images selected with respect to class, scene type, source, and visual difficulty? Were the expert annotators shown the ground-truth class labels when preparing the reference captions, particularly those containing scientific names?

5 comments
AuthorsOpenReview ↗

Response to Reviewer yATm (1/4)

We thank the reviewer for the detailed feedback and for recognizing the dataset's provenance and evaluation coverage. We address W1–W2 and Q1–Q4 below, with supporting analyses in Appendix G.1–G.6 of the revised manuscript.

W1. Species-label verification and its implications for the benchmark.

We have now added a label audit by a second expert on 100 images randomly sampled from the full dataset. The initial review flagged nine cases for closer taxonomic examination. Targeted re-examination confirmed the existing species labels in all nine cases. No species-label errors were identified in the 100-image sample after this two-stage audit.

This adds direct sample-level verification to the original curation in Appendix A.2; Q1 details the original reviewer's background and checks. The audit is distinct from an inter-annotator agreement study. We will publish the annotation protocol on Hugging Face and use public Issues to collect and respond to label feedback. We will review reported errors and document confirmed label corrections in subsequent dataset releases.

W2. Synthetic RGB sensitivity versus real underwater acquisition changes.

The Jerlov conditions are controlled, marine-inspired RGB diagnostics, as described in Appendix F.1. Their purpose is to compare checkpoint sensitivity under specified changes. They do not implement depth-resolved radiative transfer or test the combined effects of attenuation, backscatter, uneven illumination, occlusion, and clutter in real underwater scenes.

We expanded both original sensitivity panels to seeds 41, 42, and 43, covering eight distinct models. We also added 10,000-resample, within-class observation-group bootstrap intervals. Sampling intervals are conditional on the evaluated checkpoints; training-seed SD is reported separately. For example, DINOv2's Jerlov-II Top-1 drop is 10.88 percentage points (95% sampling interval: 10.07–11.69; seed SD: 0.55). Appendix G.4 reports per-class and scene uncertainty in Tables 30–31 and three-seed sensitivity in Figure 25.

To complement this diagnosis with real photographs, we added a source-platform holdout of 1,033 images, 1,016 observation groups, and 86 species. Training uses 8,264 iNaturalist images across all 100 classes; 1,033 separate iNaturalist images provide validation. All partitions are observation-group-disjoint. An independent iNaturalist test contains 3,281 images from the same 86 species, outside the nominal 8:1:1 allocation. ResNet-50 and DINOv2 each use the original fixed recipe for 100 epochs with seeds 41 and 42. The same selected checkpoint evaluates both test domains, retaining full 100-class predictions.

Macro-F1 averages over the shared 86 true classes. Values are percentages: mean ± seed sample SD [95% sampling interval]. Intervals use 10,000 within-class observation-group bootstrap resamples of the two-seed mean.

ModelIndependent iNaturalist testHeld-out platforms
ResNet-5063.88 ± 0.28 [62.24, 65.31]39.96 ± 2.56 [37.24, 41.96]
DINOv2 ViT-B/1477.73 ± 0.73 [76.25, 79.02]54.83 ± 1.66 [52.22, 57.41]

The decline occurs for both models and both seeds. It also persists within controlled and intertidal scenes and among the 44 species with at least ten held-out groups. This provides complementary evidence of acquisition transfer across recorded platforms and their image mixtures. Appendix G.6 gives the partitions in Table 33, both-domain Macro-F1 and balanced accuracy in Table 34/Figure 27, and the predefined subset checks in Table 35/Figure 28.

Q1. What qualifications and taxonomic resources were used, and how many labels were corrected, rejected, or ambiguous?

The original reviewer was a student in aquatic sciences with relevant domain knowledge and taxonomic experience. The reviewer checked each image's subject against its source label. WoRMS accepted names and synonym mappings were used to normalize taxonomic names. Records with conflicting or unresolved species identities were excluded before subsequent processing.

Separate historical totals for species-label corrections, taxonomic rejections, and ambiguous cases cannot be reliably reconstructed from the retained records. The available screening records document 1,075 content exclusions and 1,097 duplicate-related excluded images. The latter comprise 226 strict-duplicate non-retained images and 871 conservatively excluded suspected duplicates. Appendix A.2 and Table 4 describe the curation and duplicate audit; W1 reports the newly completed second-expert label audit.

AuthorsOpenReview ↗

Response to Reviewer yATm (2/4)

Q2. How much photographer, location, time, and platform overlap remains across the original partitions?

The original partitions are observation-group-disjoint, while photographer strings, recorded platforms, and observation years overlap. We quantified these overlaps from the released metadata.

After Unicode normalization, whitespace trimming, and case folding, training/validation share 1,043 photographer strings, training/test share 1,005, and validation/test share 661. There are 649 strings present in all three partitions. These are recorded-name matches, rather than verified identities across platforms.

The recorded source-platform counts are:

PlatformTrainingValidationTest
iNaturalist31,3143,9223,861
GBIF7539988
Wikimedia Commons6176
iDigBio1112
EOL401

Observation dates are available for 2,045/40,130 images (5.10%). After excluding three upload-date entries, 2,042 records provide parseable observation years. Training/validation, training/test, and validation/test share 16, 13, and 7 distinct observation years, respectively. The metadata lacks latitude, longitude, and location fields, so geographic overlap cannot be quantified from this release.

We also added a split with no cross-partition observation groups or normalized photographer strings. It is compared with a group-only control matched in per-class image budgets; both tests contain 4,023 images and all 100 classes. Three models use the original fixed recipes with two seeds per design. Macro-F1 is mean ± sample SD (%):

ModelGroup-only controlPhotographer-string-disjoint
ResNet-5079.70 ± 0.0376.46 ± 0.84
MaxViT-Tiny85.99 ± 1.1182.96 ± 0.04
DINOv2 ViT-B/1483.13 ± 0.6881.47 ± 0.81

These results extend evaluation across acquisition conditions for all 100 species. The two designs use different test photographs, so their scores reflect the combined acquisition differences. Appendix G.5, Table 32 and Figure 26 provide the design and separate sampling intervals. The platform holdout in W2 removes recorded-platform overlap with training and validation. Its test includes GBIF (940), Wikimedia Commons (74), iDigBio (14), and EOL (5). No training/validation image matched these held-out images by exact hash or perceptual hash within Hamming distance 4 (Appendix G.6).

AuthorsOpenReview ↗

Response to Reviewer yATm (3/4)

Q3. Are the anchor-based recipes adequate for other architectures, and could the outliers reflect unsuitable settings or incomplete convergence?

The new checks show that training settings contribute to the observed performance. The original anchor search selected shared optimizer profiles rather than separate optima for all architectures. MobileNetV3-Large and ViT-B/16 each used six learning-rate (LR)/weight-decay candidates, evaluated for 30 epochs at seed 41 by validation Macro-F1. The resulting SGD and AdamW profiles were used for the 23-model, three-seed, 100-epoch benchmark (Appendix B.1, Tables 5–6). The separate 80-backbone, single-seed, 30-epoch scan gives broader descriptive coverage under one recipe (Appendix B.4).

We tested the original top six and both outliers at {1/3,1,3}\{1/3,1,3\} times each model's original LR. Each run used 100 epochs, 224-pixel inputs, batch size 48, and the official splits. Seed-41 validation Macro-F1 selected the LR; the selected recipe was then run with seeds 42 and 43. Checkpoints were also selected by validation Macro-F1. This gives 40 completed runs: 24 LR checks and 16 additional-seed runs.

Top-1 accuracy is mean ± sample SD across three seeds (%):

ModelSelected LR factorSubmitted protocolAdditional protocol
DINOv2 ViT-B/141/382.71 ± 0.9487.56 ± 0.28
MaxViT-Tiny384.39 ± 0.2286.01 ± 0.37
RegNetY-8GF183.33 ± 0.4783.35 ± 0.57
ConvNeXt-Tiny382.10 ± 0.4282.87 ± 0.09
EfficientNetV2-S182.35 ± 0.2982.33 ± 0.37
Swin-Tiny181.67 ± 0.4381.62 ± 0.35
RepViT-M1.01/351.37 ± 2.6361.01 ± 3.59
SwinV2-Tiny349.08 ± 6.6453.19 ± 2.02

The exact top-six ordering changes: three of 15 pairwise orderings reverse (Kendall τ=0.60\tau=0.60), with DINOv2 replacing MaxViT at the top. Within these eight models, the original top six remain ahead of both outliers. This table separates the submitted and additional protocols; LR effects are evaluated with matched controls in the same additional training environment. At seed 41, the selected LR improves Top-1 by 4.70 points for RepViT, 4.98 for DINOv2, and 1.29 for MaxViT. Appendix G.2 provides the complete validation grid (Table 26), three-seed results (Table 27), and matched comparisons (Figure 23).

The curves also distinguish SwinV2's instability from slow convergence. Its selected runs peak in validation Macro-F1 at epochs 7/14/7, followed by increasing training loss and declining validation performance. At seed 41, one-third LR delays the peak from epoch 7 to epoch 50. Peak/final validation Macro-F1 changes from 32.67/5.68% to 48.85/40.39%. This lower-LR diagnostic reduces the subsequent decline; the validation-selected LR factor remains 3. Rising training loss after an early peak points to recipe-dependent optimization instability, rather than simply an insufficient epoch budget. Figure 24 shows unsmoothed curves, and Table 28 reports peak and final values. The new evidence supports comparing specified model–training configurations rather than treating the leaderboard as an architecture-optimal ordering.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.