acceptodds
Withdrawn from ICLR 2027

TaxaWalk: Species Recognition as a Walk Down the Tree of Life

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Biodiversity has long been organized through a hierarchical system of biological classification. This hierarchy is closely related to visual recognition from broad to fine-grained taxonomic distinctions, narrowing the space of possible identities at each rank. Yet existing methods either classify species directly or predict each taxonomic rank independently, without allowing coarse-rank decisions to guide finer-grained identification. We propose a different perspective, modelling identification as a traversal of the taxonomy rather than a classification over its leaves. We introduce TaxaWalk, a hierarchical taxonomic model that autoregressively predicts an organism's lineage, one rank at a time, conditioned on preceding ranks. This formulation enables expert-conditioned inference where a biologist can specify a taxon at any rank, and the model completes the remaining lineage consistently. We further introduce an adaptive training objective that focuses learning on underperforming ranks, improving fine-grained recognition without allowing easier, coarse-grained predictions to dominate training. TaxaWalk sets a new state of the art in zero-shot open-vocabulary recognition across multiple biodiversity benchmarks, while supporting expert conditioning and zero-shot novelty detection. All code and models will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Representation Learning, Ecology, Hierarchical Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_AhGWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TaxaWalk for hierarchical species recognition. TaxaWalk predicts the text embedding of the next rank based on the image and the predicted embedding of the previous rank. TaxaWalk was evaluated on five settings and achieved better performance compared to existing methods.

Critical strengths

The paper is well-written and well-motivated. Modeling taxonomy as autoregressive traversal for species recognition is a novel approach. This architecture also allows expert knowledge to be incorporated at inference time, which is very useful for many ecological applications. It was evaluated on a new dataset that this paper introduced - iNat Co-Observed. Different from most existing species recognition datasets which assume there is only one species in the image, iNat Co-Observed is a great evaluation dataset to evaluate whether conditioning predictions on higher taxonomic hierarchy is better than masking. TaxaWalk shows that the autoregressive approach is better than simply masking certain taxa using expert knowledge.

Critical weaknesses
  1. Questions on results

    Open-vocabulary species recognition: For TaxaWalk AR and EC, why is the mean-across-ranks top-1 accuracy lower than the species top-1 accuracy for Camera Trap, Plankton, Insects-2 and Rare Species? This is worth further discussion.

    Zero-shot novel species detection: Many “novel” species in the Open-Insect dataset overlap with TreeOfLife-200M, so this task should not be called “zero-shot” novel species detection. A more appropriate name is needed. The MSP baseline can be extended to the genus and family levels by summing softmax scores across the corresponding genera or families. Table 5 should also include these results for better comparison. Similarly, for local and non-local, AUROC on the genus or family levels are so much lower compared to the species level. Why? Especially for C-Am. local, the family level AUROC (48.84 ~= 50) is the same as a random binary classifier. Similarly, this is worth further discussion.

  2. What is hierarchical pruning? The paragraph on closed-set comparison is difficult to read (Lines 314 - 323).

Questions for authors
  • The results on few-shot classification should be briefly discussed in the main text.

Minor issues

  • Table 1 and Table 2: Grouping by the model architecture will make the presentation clearer. ViT-B (BioCLIP, …, RCME), ViT-L (CLIP, …, BioCLIP-2, TaxaWalk (AR)), and TaxaWalk (EC) should be in its own row as it gets additional guidance at inference time. It’s clearer to show that TaxaWalk (AR) outperforms other methods and TaxaWalk (EC) can further improve TaxaWalk (AR).
  • Line 167 “in the rollout (AR)” -> in the [autoregressive] rollout (AR)
  • Line 244 - no citation for “the GBIF Backbone Taxonomy”
  • Cite the published version of CrypticBio, not the arxiv version
  • Figure 1: define R_t in the figure caption
Reviewer_5PnmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TaxaWalk, an open-vocabulary species recognition framework that predicts a taxonomic lineage autoregressively, one rank at a time. On top of BioCLIP-2, a small recurrent predictive module predicts the text embedding of the next rank given the image and the ranks already decided; a MoRE head and a rank-balanced training objective round out the design. Because prediction is sequential, a taxonomic label supplied by an expert at any rank steers everything predicted below it. The evaluation spans open-vocabulary and cryptic species recognition, expert-conditioned identification, and novelty detection on a range of biodiversity benchmarks. The paper also contributes iNat Co-Observed, a multi-species benchmark built around taxonomic hints. TaxaWalk achieves substantial improvements on several species recognition benchmarks and shows gains in expert-conditioned identification and novelty detection.

Critical strengths

1.Unlike existing flat recognition methods, TaxaWalk makes direct use of taxonomic dependencies at prediction time while keeping the candidate vocabulary open — to me this is a sensible way to reconcile hierarchical modeling with open-vocabulary recognition, and the numbers back it up. The model attains the best species-level accuracy on every one of the seven benchmarks in Table 1, and the margins on Camera Trap, Plankton, and Insects-2 are especially striking. The CrypticBio results add further evidence that the approach helps in fine-grained regimes. I also appreciate that the paper goes beyond headline numbers: the analyses of hierarchical pruning, embedding regression, and the architectural ablations give useful supporting evidence across diverse settings.

2.Being able to condition on expert-provided taxonomic information is practically appealing, since in many real settings a biologist already knows part of the answer. The newly introduced iNat Co-Observed benchmark tests this on roughly 6,000 multi-species photographs across different hint depths. I found the comparisons carefully set up: every method receives the same hints, the contrastive baselines run with their stronger complete-lineage retrieval procedure, and Appendix E.2 further separates shared from distinct hints. The improvements hold across these variants, which I take as solid evidence that expert-conditioned identification is useful beyond standard image-only recognition.

Critical weaknesses

1.My main concern is that the large gains over BioCLIP-2 are not cleanly attributable to autoregressive rank conditioning. TaxaWalk differs from the backbone in several ways at once — additional training on TreeOfLife-200M, LoRA adaptation, and the added prediction modules — and any of these could account for part of the improvement. Table 3 and Figure 4 probe a number of design choices, but neither includes a non-autoregressive rank-wise baseline trained under comparable conditions. Without such a control, it is hard to tell how much of the gain comes from the paper's central modeling idea rather than from stronger training overall.

2.Section 4.4 reports per-rank novelty detection AUROC, but it does not directly evaluate whether the model can pinpoint the earliest novel rank — arguably the more interesting claim attached to Figure 1c. A related issue shows up in Appendix C: in the CONGENER setting the first incorrect rank is species by construction, so a trivial always-species predictor would reach 100% localization accuracy, which makes the quoted 16.7% chance baseline uninformative there. As it stands, I don't think the rank-localization claim is adequately supported.

Questions for authors
  1. Could you compare TaxaWalk against a non-autoregressive rank-wise embedding predictor trained under comparable conditions? Even a smaller-scale version would help. A control like this would clarify what autoregressive conditioning itself contributes to the recognition improvements.
  2. Concretely, how would the rank-specific novelty scores be used to identify the earliest unknown rank for an individual specimen, and could you evaluate this capability directly? Relatedly, could you comment on the chance-level comparison in Appendix C's CONGENER setting, where the true divergence rank appears to be fixed at species, and report an appropriate baseline for the RANDOM setting?
Reviewer_UuvaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors use low rank Fine-Tuning on the visual encoder of bioclip 2 using text strings corresponding to any rank level, rather than only the lowest available. This allows to sequentially assign an image of a specimen the label that corresponds to each rank, using the labels in each level to inform the prediction of the lower levels. They also introduce several new evaluation settings that seem very relevant to the community, and on which their method performs very well.

Critical strengths

The proposed benchmark settings seem genuinely new and relevant to the biodiversity community, such as the cooccurrence cases and the mislabeled image detection.The authors would do well putting more emphasis on them. The paper is easy to follow and the proposed approach makes sense to me.

Critical weaknesses

I am not fully convinced by the evaluation setting. If I got it we'll, comparisons are against methods that lack the additional Fine-Tuning and parameters of TaxaWalk, making the comparison inconclusive. Additional ablations, including Fine-Tuning bioclip with the same approach but without the hierarchy-aware regression and TaxaWalk but without the rank conditioning, would resolve this issue.

Questions for authors

Have the authors run the ablations:

  • Bioclip 2 Fine-Tuning with the same dataset and lora setting?
  • TaxaWalk without the conditioning on previous ranks?
Reviewer_FboTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the problem of open vocabulary, hierarchical animal taxonomy classification from images. The authors argue that current approaches either classify species directly or predict each taxonomic rank on its own. Instead, they attempt to reframe the identification as a recursive classification down the taxonomy one rank at a time. Their method builds on top of a frozen BioCLIP-2 encoder. Its image encoder is fine-tuned with LoRA, and the text encoder is kept frozen and used to embed the label names at every rank. The authors then use a transition model made of multiple GRU cells built around a transformer block conditioned on the image, followed by a mixture-of-experts output layer. At each classification level the model takes the embedding of the current rank's label, the image embedding and a rank embedding, and then encodes and outputs the next rank. To select a label a set of candidate labels are also encoded and the classification is done based on the shortest distance to the candidate labels. This distance is also used as a score for whether the specimen is novel at that rank. During training the model is always given the true previous label, and the loss is a weighted maximum over the per-rank errors. At test time they report two modes, AR, where the model starts from the image alone and feeds its own predictions forward, and EC, where it is given the true ancestors. The authors evaluate on multiple open vocab benchmarks and show large performance improvements for each.

Critical strengths
  • Overall the method is simple and greatly improves on a well-known model and new species can be added without retraining
  • The model is also able to give a per rank novelty score which could be an important biodiversity indicator.
Critical weaknesses
  • Certain phrases in the abstract and introduction about the current hierarchical classification field are misleading. The abstract states that "either classify species directly or predict each taxonomic rank independently" which is not true and is inconsistent with the paper's own related works section with the top-down classifiers.
  • The novelty of the work is also overstated. Many previous approaches decode the label sequence with an RNN over image features including CNN-RNN [1], Koo et al. [2] and Condition-CNN [3]. In the open-vocabulary setting we also have WildMatch [4] and Ren et al. [5], which also walk down a hierarchy and restrict candidates to the predicted subtree. Regressing into a fixed text embedding and decoding by nearest neighbor is DeViSE [6]. The loss is tilted ERM [7] applied over ranks which the paper cites but the abstract still lists the objective as a contribution.
  • It seems that experts are selected from the hidden state of batch mean, so at test time one image's prediction depends on the other images. The paper also says the router ends up on a fixed pair of experts per rank, which makes MoRE a per-rank head, and that is what Figure 4d should compare against rather than a dense head.
  • Knowing that, it would have been interesting to see the std of multiple training or even just test seeds to see the effect of this on the actual predictions. It is also useful in general since some results have smaller margins especially in table 11
  • There should also be a control to determine whether the gains come from the traversal or from the extra training. TaxaWalk gets a LoRA pass over TreeOfLife-200M with lineage supervision and a decoder of up to 368M parameters. On the other hand, every baseline in Tables 1, 2 and 5 is a frozen zero-shot model. Also, all of the ablations in Figure 4 are run under EC that give the model the true ancestor, so none of them tells us anything about the AR setting that the main claims rest on.
  • As another minor issue, the text needs some improvement to remove long run on sentences and some weird formatting with bolds and italics.

[1] Y. Guo, Y. Liu, E. M. Bakker, Y. Guo, M. S. Lew. CNN-RNN: a large-scale hierarchical image classification framework. Multimedia Tools and Applications, 2018.

[2] J. Koo, D. Klabjan, J. Utke. Combined Convolutional and Recurrent Neural Networks for Hierarchical Classification of Images. arxiv 2018.

[3] B. Kolisnik, I. Hogan, F. Zulkernine. Condition-CNN: A hierarchical multi-label fashion image classification model. Expert Systems with Applications, 2021.

[4] Z. Fabian et al. Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images. arxiv 2023.

[5] Z. Ren, Y. Su, X. Liu. ChatGPT-Powered Hierarchical Comparisons for Image Classification. NeurIPS, 2023.

[6] A. Frome et al. DeViSE: A Deep Visual-Semantic Embedding Model. NeurIPS, 2013.

[7] T. Li, A. Beirami, M. Sanjabi, V. Smith. Tilted Empirical Risk Minimization. ICLR, 2021

Questions for authors
  1. Does inference depend on batch size because of the batch-mean routing? What happens at batch size 1?
  2. The Table 1 the caption states that ranks are scored on different image sets, which is why species accuracy can exceed mean-rank accuracy. However, this makes it hard to see where errors happen in the AR rollout. Could you report per-rank accuracy on a shared image set? Additionally, on Rare Species BioCLIP-2's mean-rank accuracy is higher than TaxaWalk's (80 vs 78.8). Which ranks account for this?
  3. Did you run multiple seeds to see performance stability?
  4. Were the Table 1 benchmarks and CrypticBio deduplicated from TreeOfLife-200M?

I would be willing to lift my score if the authors address some of the control concerns in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.