- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MetaFauna, an animal re-identification dataset containing 44,317 images of 2,500 individuals across eight species. It combines three existing data sources, adds identity annotations for Snapshot Safari, and provides structured metadata describing visibility, capture conditions, and environmental context. The paper also proposes MetaRouter, which uses metadata to route LoRA experts within a frozen DINOv3 backbone. Experiments across eight species, five training seeds, and controlled ablations show improvements over visual-only and metadata-aware baselines. Additional experiments examine missing metadata and joint multi-species training.
Critical strengths
MetaFauna provides image-level descriptions of which identity cues are visible and how clearly they are captured. This supports a useful research question that identity labels alone cannot directly address: how observation conditions should guide animal identification. The consistent schema and source mappings strengthen its value as a reusable benchmark. The attribution experiments are stronger than a simple comparison against external baselines. Static expert mixtures, visual routing, feature fusion, and shuffled metadata help separate the contributions of capacity, routing, and image–metadata correspondence. MetaRouter consistently improves over these controls across five seeds, providing meaningful evidence for metadata-conditioned adaptation.
Critical weaknesses
The new identity annotations lack a quantitative reliability assessment. Appendix A describes multiple annotators and cluster-level review, but reports neither agreement nor an independent audit of identity merges and splits. Reviewing likely matches can validate accepted associations while still missing links between separate identity clusters. Such missed links could place the same animal in nominally identity-disjoint partitions. This uncertainty directly affects the dataset’s central contribution and the validity of its evaluation. The protocols do not establish how well the benchmark measures identification across independent encounters. Snapshot Safari separates query and gallery by source sequence, but does not report their separation by camera site or capture time. Hyena and Leopard use image-level splitting because encounter information is unavailable. Near-duplicate or same-encounter matches therefore remain a possible shortcut. The reported gains establish performance on these partitions, but provide limited evidence of robustness across encounters.
Questions for authors
-
What existing annotation-quality statistics are available, such as agreement rates, unresolved-pair rates, or independent identity audits? How were potentially missed associations outside the highest-ranked candidates checked?
-
Were near-duplicate or same-encounter images audited across the Hyena and Leopard query/gallery partitions? Please summarize any existing checks.
-
How many query identities in the Snapshot Safari subsets have gallery matches from different camera sites or substantially separated capture times? This would clarify the deployment setting represented by the benchmark.
6 comments
1.2 How were potentially missed associations outside the highest-ranked candidates checked?
An independent retrieval audit of 5,781 cross-cluster candidate pairs confirmed zero train–test identity conflicts and resolved two missed test-partition associations prior to finalizing the benchmark. Specifically, during dataset construction, we used an independent visual retrieval model to identify potential same-individual associations overlooked by the original DINOv3 candidate ranking, followed by manual verification of the corresponding source sequences.
Retrieval calibration. We first evaluated whether the independent model could recover known associations that received low DINOv3 rankings. In a leave-one-encounter-out evaluation across 286 eligible cases, the model recovered the correct identity within its top-5 candidates in 245 cases (85.7%). Among the 20% of cases ranked lowest by DINOv3 (58 cases), it recovered 41 (70.7%), showing its ability to retrieve challenging associations that received low rankings from the original model.
Independent candidate audit. If a true association was missed during initial candidate screening, the same individual would remain split across two separate identity clusters. To identify such cases, we used the independent model () to retrieve additional candidate matches within and across the provisional training and test partitions, yielding 5,781 unique candidate pairs. All retrieved candidate pairs were manually reviewed using the complete source sequences and the original verification criteria. No train–test identity conflicts were confirmed among the reviewed pairs, while two missed associations were identified within the test partition and resolved before finalizing the identity mapping and evaluation splits. The dataset statistics, fixed splits, and experimental results reported in the submission correspond to this finalized benchmark, and we have documented this retrieval audit in Appendix A.1.3 of the revised PDF.
2. Were near-duplicate or same-encounter images audited across the Hyena and Leopard query/gallery partitions?
Thank you for raising this concern. Our similarity-aware audit across all 9,935 Hyena and Leopard images confirmed 655 near-duplicate or same-encounter groups, with zero confirmed groups spanning the query/gallery partitions. MetaFauna adopts identity-disjoint train/test partitions to evaluate generalization to unseen individuals. For the six Snapshot Safari species, query and gallery images are separated at the source-sequence level. Because HyenaID2022 and LeopardID2022 do not provide encounter identifiers for direct encounter-level splitting, we incorporated similarity-aware verification during split construction to prevent repeated observations from creating easy positive matches across their query/gallery partitions.
Similarity-aware verification. We examined all 9,935 Hyena and Leopard images using exact-duplicate checks, perceptual similarity across both animal crops and original frames (capturing near-identical images that differ in cropping, encoding, or minor visual details), and the similarity-aware visual clustering procedure from WildlifeReID-10k [1]. Candidate pairs were independently examined by two annotators, with a third annotator adjudicating positive, conflicting, or uncertain cases. Near-duplicates were identified based on substantially repeated visual content, while same-encounter decisions additionally considered available camera identifiers, timestamps, burst information, and scene consistency.
Audit results and split validation. Across the 9,935 images from the two source datasets, the WildlifeReID-10k visual clusters mapped to MetaFauna identify 727 candidate visual groups involving 1,982 images from 174 identities (Hyena: 132 groups, 322 images, 55 identities; Leopard: 595 groups, 1,660 images, 119 identities). Combining these groups with additional perceptual-similarity candidates, independent human review confirmed 655 near-duplicate or same-encounter groups, comprising 1,743 images from 162 identities. For test identities, each confirmed repeated-observation group was assigned entirely to either query or gallery while preserving valid positive matches. A final check found no confirmed groups spanning both partitions. We have included these similarity-aware audit and split validation details in Appendix C.1 (and the foreground-only evaluation in Appendix F.2, Table F.3) of the revised PDF.
[1]. Adam, Lukas, et al. "WildlifeReID-10k: Wildlife re-identification dataset with 10k individual animals." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.
Thank you for the constructive review and for recognizing the value of MetaFauna's observation-level metadata schema and the controlled attribution study in Sec. 4.3. The concerns focus on the quantitative reliability of the new Snapshot Safari identity annotations, including potential false merges and missed associations across partitions, and the extent to which the evaluation splits prevent same-encounter shortcuts and evaluate identification across independent encounters. We address each point below through: (1) species-level inter-annotator agreement statistics, adjudication outcomes, and an independent audit of 270 identity clusters (Q1.1); (2) a calibrated retrieval-based audit of 5,781 cross-cluster candidate pairs confirming zero train–test identity conflicts among the reviewed pairs (Q1.2); (3) a similarity-aware audit of all 9,935 Hyena and Leopard images confirming that none of the 655 near-duplicate or same-encounter groups spans the query/gallery split (Q2); and (4) a spatial and temporal breakdown of query–gallery separation across the Snapshot Safari subsets (Q3).
1.1 What existing annotation-quality statistics are available, such as agreement rates, unresolved-pair rates, or independent identity audits?
Thank you for highlighting the importance of quantifying annotation reliability. Across the six Snapshot Safari species, initial inter-annotator agreement reaches 91.0% (κ=0.825), and an independent audit of 270 identity clusters confirmed zero incorrect identity merges. Below we provide additional agreement statistics and an independent identity audit for the six Snapshot Safari species whose individual identities were annotated in MetaFauna. Hyena and Leopard retain their source-provided identity labels and are therefore excluded from these analyses.
Annotation agreement. As described in Appendix A.1.2, two annotators independently classify candidate sequence-group pairs as same, different, or unresolved. A third annotator reviews all proposed positive associations and cases involving disagreement or uncertainty, and only same-individual associations confirmed by all three annotators are accepted.
Across 30,217 reviewed candidate pairs, the initial agreement rate is 91.0%, with Cohen's κ=0.825 and 91.9% positive agreement (PA) on same-individual decisions. The unresolved-pair rate is 4.4%, defined as the proportion of pairs for which at least one initial annotator selected unresolved. The species-level statistics are summarized below.
Inter-annotator agreement across Snapshot Safari subsets. Entries report pair-level agreement statistics prior to third-annotator adjudication.
In total, 14,770 candidate pairs underwent third-annotator review, comprising 11,880 pairs initially marked same by both annotators, 2,732 initial disagreements, and 158 pairs marked unresolved by both annotators. Following adjudication, 11,482 associations were confirmed as same, 2,784 were classified as different, and 504 remained unresolved. Unresolved pairs were never used to establish identity associations.
Independent identity audit. In addition to the cluster-level consistency checks performed during annotation, we independently examined 270 identity clusters (45 per species), covering 7,140 images and 2,720 source sequences. These audited clusters include identities with previously disputed associations, large clusters, and identities spanning different cameras or capture dates, alongside randomly selected remaining identities. Independent reviewers inspected the complete sequences and persistent individual-specific visual features without access to the original similarity scores or identity decisions. No incorrect identity associations were confirmed among the audited clusters. We have added the agreement statistics, adjudication outcomes, and independent audit details to Appendix A.1.3 (Table A.1) of the revised PDF.