- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces TaxaWalk for hierarchical species recognition. TaxaWalk predicts the text embedding of the next rank based on the image and the predicted embedding of the previous rank. TaxaWalk was evaluated on five settings and achieved better performance compared to existing methods.
Critical strengths
The paper is well-written and well-motivated. Modeling taxonomy as autoregressive traversal for species recognition is a novel approach. This architecture also allows expert knowledge to be incorporated at inference time, which is very useful for many ecological applications. It was evaluated on a new dataset that this paper introduced - iNat Co-Observed. Different from most existing species recognition datasets which assume there is only one species in the image, iNat Co-Observed is a great evaluation dataset to evaluate whether conditioning predictions on higher taxonomic hierarchy is better than masking. TaxaWalk shows that the autoregressive approach is better than simply masking certain taxa using expert knowledge.
Critical weaknesses
-
Questions on results
Open-vocabulary species recognition: For TaxaWalk AR and EC, why is the mean-across-ranks top-1 accuracy lower than the species top-1 accuracy for Camera Trap, Plankton, Insects-2 and Rare Species? This is worth further discussion.
Zero-shot novel species detection: Many “novel” species in the Open-Insect dataset overlap with TreeOfLife-200M, so this task should not be called “zero-shot” novel species detection. A more appropriate name is needed. The MSP baseline can be extended to the genus and family levels by summing softmax scores across the corresponding genera or families. Table 5 should also include these results for better comparison. Similarly, for local and non-local, AUROC on the genus or family levels are so much lower compared to the species level. Why? Especially for C-Am. local, the family level AUROC (48.84 ~= 50) is the same as a random binary classifier. Similarly, this is worth further discussion.
-
What is hierarchical pruning? The paragraph on closed-set comparison is difficult to read (Lines 314 - 323).
Questions for authors
- The results on few-shot classification should be briefly discussed in the main text.
Minor issues
- Table 1 and Table 2: Grouping by the model architecture will make the presentation clearer. ViT-B (BioCLIP, …, RCME), ViT-L (CLIP, …, BioCLIP-2, TaxaWalk (AR)), and TaxaWalk (EC) should be in its own row as it gets additional guidance at inference time. It’s clearer to show that TaxaWalk (AR) outperforms other methods and TaxaWalk (EC) can further improve TaxaWalk (AR).
- Line 167 “in the rollout (AR)” -> in the [autoregressive] rollout (AR)
- Line 244 - no citation for “the GBIF Backbone Taxonomy”
- Cite the published version of CrypticBio, not the arxiv version
- Figure 1: define R_t in the figure caption