acceptodds
Under review as a conference paper at ICLR 2027

PROTAX-SSL: Probabilistic Taxonomic Classification with Self-Supervised Representations

Abstract

Taxonomic classification of DNA barcodes is fundamental to biodiversity monitoring, yet remains challenging due to class imbalance, incomplete databases, and unknown taxa. In such settings, reliable taxonomic inference requires calibrated uncertainty estimates and identifying unknown taxa. Self-supervised foundation models have demonstrated high-order representations that capture taxonomic relationships, but downstream classifiers built on them typically lack principled uncertainty estimates and mechanisms for modeling unknown taxa. PROTAX, a probabilistic classifier from statistical ecology, provides calibrated predictions and explicitly models taxonomic structure complexities, but operates on raw sequences, limiting its accuracy relative to self-supervised approaches. We bridge these two lines of work with PROTAX-SSL, which couples self-supervised DNA barcode representations with probabilistic taxonomic inference. We evaluate variants of PROTAX-SSL across three datasets spanning diverse experimental settings. PROTAX-SSL improves calibration and unknown-species detection, while its modular design leverages complementary representations to improve accuracy. We provide a basis for reliable taxonomic analysis and potential species discovery.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Probabilistic Models, Self-supervised Representations, Uncertainty Calibration, Taxonomic Classification, Open-set Recognition, Multi-representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q7vqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PROTAX-SSL integrates frozen BarcodeBERT and BarcodeMamba representations into the hierarchical classification framework of PROTAX-GPU. Embedding distances supply taxon-level similarity features, with linear and MLP variants supporting single or combined representations. Evaluation covers fungal and arthropod datasets, predictive calibration, and detection and genus-level placement of purported unknown species. Results show substantial gains over raw-sequence PROTAX on challenging fungal data, improved calibration over similarity-based confidence, and smaller or mixed benefits on curated arthropod data.

Critical strengths

The paper addresses a useful gap between strong barcode representations and reliable taxonomic decisions. Combining learned representations with a classifier that explicitly models taxonomic hierarchy and unknown branches provides a practical approach to incomplete reference databases. The fungal results show that this combination can improve accuracy while producing more reliable confidence estimates. The evaluation includes two encoder architectures and both in-domain arthropod and out-of-domain fungal data. The FinBOL results also show where learned representations offer little benefit over raw sequence distances. This comparison helps establish the conditions under which the proposed combination is useful.

Critical weaknesses

The unknown-species evaluation does not establish that its test examples belong to species absent from the reference library. Section 4.1 selects sequences annotated only to genus level and treats them as unknown species. However, missing species annotations do not establish biological novelty: these sequences could belong to species already represented in the library. Selecting different sequences from the same genus also does not establish species separation between the unknown training and test examples. This leaves the reported 99.9% detection accuracy difficult to interpret as evidence of unseen-species detection.

The calibration comparisons do not isolate the benefit of PROTAX from ordinary score calibration. The encoder baselines use cosine similarity as a proxy for probability, while PROTAX learns a supervised mapping from distance features to probabilities. The paper does not compare against validation-calibrated distance-based classifiers or comparable rejection rules. Since the underlying PROTAX framework already supports heterogeneous predictors, stronger controls are needed to establish what the proposed integration adds beyond improved features and a learned confidence mapping.

Questions for authors
  1. How was species absence verified for the genus-only sequences used as unknown examples? Can the authors provide an evaluation that withholds fully identified species from both classifier training and the reference library?

  2. Does Table 2 measure detection on unknown queries alone or on a mixture of known and unknown queries? Please report the known-species false-positive rate and clarify whether placement accuracy is conditional on successful detection.

  3. How does PROTAX-SSL compare with a validation-calibrated classifier using the same embedding distances, including a distance-based unknown-rejection rule? This comparison would help separate the benefits of hierarchical modeling from those of supervised calibration.g the complete taxonomic path.

4 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

We thank the reviewer for the careful and constructive review. Following the reviewer’s concerns, we have replaced the unknown-species experiment with a more rigorous protocol and added a calibrated k-NN baseline with a rejection threshold, as summarized in our general response. We address each point below.

RE1: Unknown Species Evaluation Setup

We thank the reviewer for this important point. We agree that genus-only annotations do not guarantee that a sequence belongs to a species absent from the reference library. Since our problem statement and motivation center on detecting novel species, we have replaced this experiment with a setup in which unknown species are simulated by withholding whole species from the training data and the reference library.

Please refer to our general response above for the experiment details and results. The new protocol is added to Sections 4.1 and 5.3 in the revised manuscript.

RE2: New Unknown Species Evaluation Results:

We train the PROTAX models and the k-NN-calibrated models (described in our next response and Appendix H) on the unknown species training split. False-positive rates (FPRs) for unknown species detection are measured on the known queries, evaluated with self-match exclusion: for a given FPR, the detection threshold is set so that this fraction of known queries is wrongly flagged as unknown. As shown in Appendix D, self-match exclusion yields the same results as fully held-out evaluation.

Detection rates in Table 3 are reported on the unknown species test sequences only, all of which belong to species absent from both training and the reference library. Genus Placement accuracy is reported on the same sequences and is independent of detection, so it does not depend on the chosen FPR. AUROC, computed over both known and unknown queries, likewise summarizes detection independently of any FPR threshold. ROC curves are reported in Appendix I.

Table 3 results now show lower absolute detection rates when compared to the original experiment, reflecting a more realistic and generalized assessment of unknown species detection under this robust protocol. However, our main conclusions still hold: PROTAX-SSL variants outperform PROTAX-GPU, PROTAX-SSL outperforms the k-NN-calibrated + rejection models built on the same embeddings, and the multi-embedding variants perform best across all metrics. AUROC increases from 74.4 for PROTAX-GPU and 73.2 for the best k-NN-calibrated model to 80.8 for PROTAX[BarcodeBERT+Mamba, MLP].

ModelDet. @ FPR=5%Det. @ FPR=10%Det. @ FPR=15%Det. @ FPR=20%Placement (%)AUROC
BarcodeBERT (k-NN-calibrated)11.522.432.041.149.869.4
BarcodeMamba (k-NN-calibrated)25.539.148.656.572.773.2
PROTAX-GPU10.121.735.247.149.574.4
PROTAX[BarcodeBERT]12.825.637.949.550.075.2
PROTAX[BarcodeMamba]27.6‾\underline{27.6}43.554.262.573.179.3
PROTAX[BarcodeBERT+Mamba]27.6‾\underline{27.6}44.7‾\underline{44.7}56.3‾\underline{56.3}64.8‾\underline{64.8}73.4‾\underline{73.4}80.4‾\underline{80.4}
PROTAX[BarcodeBERT+Mamba, MLP]30.346.757.665.774.780.8
AuthorsOpenReview ↗

Rebuttal (Part 2)

RE3: How does PROTAX-SSL compare with a validation-calibrated classifier using the same embedding distances, including a distance-based unknown-rejection rule?

To answer this, we implemented a k-NN-calibrated model as an additional baseline. It uses the same embeddings, training data, and supervision as PROTAX-SSL, so any difference between the two can be attributed to the PROTAX framework.

At each taxonomic level, every candidate taxon is scored by its distance to the query, defined by the taxon’s closest reference sequence, which mirrors PROTAX’s minimum-distance feature. These distances are converted into probabilities by a softmax with a learned temperature for that level, and each level is calibrated independently. At the species level, we add an unknown-species option with a learned distance τ. The temperature and τ are fitted by minimizing the negative log-likelihood of the correct answers. All details are in Appendix H.

Calibration curves and ECE of this model are added to Figure 2 and compared with the base k-NN and PROTAX models in Section 5.2. The k-NN-calibrated models improve calibration considerably over the standard ones, but they generally remain behind PROTAX-SSL, which indicates that PROTAX-SSL learns more than a simple mapping from similarity scores to probabilities.

Unknown species detection results are reported in the previous answer (RE2).

RE4: PROTAX-SSL Contribution

PROTAX-SSL learns a probabilistic decision model. It combines distance-based features with features describing the reference taxonomy. The distance features capture more than the query’s proximity to a taxon: carefully designed features such as the gap between the closest and second-closest distances describe how decisive that proximity is. PROTAX-SSL then learns the reliability of each feature, and how different combinations of feature values map to correct predictions at each level. The hierarchical structure adds more. At each node, probability is distributed only among the children of the selected parent, including an unknown branch, rather than across all classes in a taxonomic level as in typical classifiers. Probabilities then propagate down the hierarchy, so uncertainty at a higher level carries into the levels below it.

Finally, while uncertainty estimation is a rich topic in machine learning, it remains largely unaddressed in taxonomic classification with barcode foundation models. To the best of our knowledge, this is the first work to address the uncertainty of barcode foundation models, providing both an analysis of their calibration and a method to improve it. It is also, to our knowledge, the first to combine multiple barcode foundation models, and as shown in our response to reviewer LRnR (RE3) and in Appendix G, simple fusion of their embedding spaces degrades performance.

The results you provided demonstrate an improvement in the ability to detect unknown species, but I am still concerned about the trade-off between computational cost and performance. Specifically, given the modest AUROC improvement from 80.4 to 80.8 when moving from the non-MLP dual-embedding variant to the MLP variant, is the additional training cost justified for such a small gain?

We thank the reviewer for the follow-up. We agree that the AUROC gain of the MLP variant is small. AUROC aggregates detection over the entire range of FPR thresholds, so differences at individual thresholds are diluted in the overall score. At fixed FPRs of 5% and 10%, the MLP variant shows larger gains in detection. Independently of any detection threshold, it also achieves the highest genus-level placement accuracy for unknown species. Its gains are also larger in species-level classification accuracy in the MycoAI experiments (Section 5.1) and in calibration, most notably at the species level (Section 5.2).

As our results show, no single PROTAX-SSL variant performs best in all cases, so the value of the trade-off between cost and performance depends on the use case. We therefore recommend the MLP variant when unknown species must be detected at low false-positive rates or placed in their correct genus, and for applications where species-level accuracy and calibrated uncertainty matter most.

Regarding cost, training is a one-time expense, and as explained in our response to Reviewer LRnR (RE4), much of the multi-embedding variants’ longer training time reflects GPU memory limits on batch size rather than model complexity. At inference, the linear and MLP dual-embedding variants run in almost the same time, since they share the encoders and distance computations.

Finally, the PROTAX[BarcodeBERT+Mamba, MLP] variant demonstrates the potential of combining complementary sources of evidence, and shows that an MLP can exploit them more effectively than a linear layer. We expect this advantage to become more pronounced with more deliberate choices of extracted features, embedding spaces or data modalities.

Reviewer_Kh6NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PROTAX-SSL to bridge PROTAX and self-supervised foundation models for DNA barcode sequences. Building on top of PROTAX-GPU, BarcodeBERT, and BarcodeMamba, PROTAX-SSL achieves better performance on classification, unknown species detection and placement, and model calibration on several fungi and arthropod datasets.

Critical strengths

This paper addresses an important problem and the proposed solution is well-motivated: PROTAX provides calibrated predictions, but can only use raw sequences; self-supervised foundation models lack uncertainty estimates. By bridging these two approaches, PROTAX-SSL achieves higher classification accuracy and better calibration. Datasets used in this paper cover two taxa for which DNA barcodes are very useful for species classification and discovery. The approach is evaluated for practical tasks including classification, unknown species detection, and uncertainty estimate.

Critical weaknesses

Sections 3 and 5 require substantial clarification. I will raise my rating if the following issues are adequately addressed.

  • The overview of the two pipelines is difficult to understand (Section 3). For example, the “reference taxonomy construction” part is very vague, and Figure 1 adds no additional information compared to the first paragraph in Section 3. How was the taxonomic tree constructed? There is no reference or details. What does “associating the resulting embeddings with their corresponding taxonomic nodes” mean? Does this mean that PROTAX-SSL uses the same taxonomic tree as PROTAX and the only difference is that PROTAX uses raw DNA barcode, while PROTAX-SSL uses features extracted by one of the four variants? Does “a post-hoc probability layer” (Line 183) refer to a linear layer or an MLP discussed in Sec. 3.2? Is it used for classification? If so, why is cosine similarity needed for classification? If not, what is it used for? Also, it is unclear which datasets were used to train the four variants. For example, in Section 5, for results presented in Table 1, were all variants trained on MycoAI, then evaluated on MycoAI, Yeast, and Filamentous? Similarly, for Table 3, were all variants trained on Canadian Invertebrates, then evaluated on Canadian Invertebrates and FinBOL?

  • The evaluation is quite extensive with several datasets and tasks, which can be a strength of this paper. However, the way it’s currently presented in Section 5 makes it quite difficult to parse. It would be so much clearer if Section 5 were structured as Sec 5.1 Taxonomic Classification Accuracy, Sec 5.2 Predictive Reliability, and Sec 5.3 Unknown Species, following Sec 4.2. Which datasets were used to evaluate unknown species detection and placement in Sec. 5.2? There’s no description in the text nor in the caption of Table 2. Why can’t BarcodeBERT or BarcodeMamba be used for kNN-based unknown-taxa detection (Sec. 5.2)? For example, use a distance threshold on the average cosine similarity to the kNNs to identify unknown tax. Was FinBOL used for training or not? Also, explain that “Can.Inv. = Canadian Invertebrates or \texttt{CanadianInvertebrates-ML}” in the caption of Table 3. Why is there no predictive reliability analysis for the arthropods datasets?

Questions for authors
  • Line 50: “but the underlying data remain difficult”. What does it mean? What’s difficult?
  • Open-set recognition vs unknown species detection: it’s better to just use one name for this task consistently. The task is mostly referred to as “unknown species/ taxa detection” throughout the paper. “Open-set recognition” is only used twice in lines 413 and 455.
  • \texttt{CanadianInvertebrates-ML} is defined in Line 250 but never used.
  • Have an additional column in Table 1 to show which models are reference-based and which are not.
  • Section 4.3 Baselines: Why are the baselines directly taken from Gao et al. (2025)?
2 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

We thank the reviewer for the thorough and constructive feedback. We are glad they found the problem important and well-motivated, and that they appreciated the evaluation. We agree that Sections 3 and 5 would benefit from clarification and address each point below.

RE1: How was the taxonomic tree constructed?

The tree is constructed exactly as in PROTAX (Somervuo et al., 2016), as described in Section 2.1. Taxonomic units at different levels are represented as nodes, and parent–child relationships define the structure. Available reference sequences are then associated with the finest-grained available taxonomic unit to which they are annotated. As a result, each node can have zero, one, or multiple sequences. An “unknown” branch is also added under each node.

We have added a clearer methodology overview in Section 3.

RE2: Does PROTAX-SSL use the same taxonomic tree as PROTAX and the only difference is that PROTAX uses raw DNA barcode, while PROTAX-SSL uses features extracted by one of the four variants?

Yes. PROTAX-SSL uses the same taxonomic tree structure as PROTAX. The two methods differ in only two respects:

  1. Sequence Representation: Reference and query barcode sequences are replaced by their embeddings from BarcodeBERT, BarcodeMamba, or both, depending on the variant.

  2. Distance features: PROTAX-GPU computes distances as p-distances between raw barcode sequences, while PROTAX-SSL computes them from cosine similarity between embeddings (Section 3.1).

These distances then populate the design matrix used for classification as described in Section 2.1.

RE3: Does “a post-hoc probability layer” refer to a linear layer or an MLP discussed in Sec. 3.2? Is it used for classification? Why is cosine similarity needed for classification?

This phrase was meant as a high-level description of the whole PROTAX-SSL framework and what it adds on top of the embedding space, not as a specific layer added to the described framework. We used “post-hoc” because the framework operates on top of the frozen encoders without modifying them, and “probabilistic” because it calibrates predictive uncertainty.

As described in our previous answer, cosine distances are extracted from the embedding space and used to populate the design matrix. The design matrix is then mapped, by either linear regression parameters or an MLP depending on the variant, to a probability distribution over the child branches at each node. We indicate in our results tables when a linear layer or MLP is used.

We agree that this phrase adds unnecessary confusion and we have removed it.

RE4: It is unclear which datasets were used to train the four variants.

For Table 1: all variants were trained on the MycoAI Benchmark, and then evaluated on the MycoAI Benchmark, Yeast, and Filamentous, all with self-match exclusion applied.

For Table 2, Canadian Invertebrates and FinBOL were used independently. On Canadian Invertebrates, all models were trained on the 70% training split of the Seen subset and evaluated on both that training split (with self-match exclusion) and the 20% test split. Both splits are described in Sec 4.1. On FinBOL, models were trained and evaluated on FinBOL, with self-match exclusion applied. So, each dataset has its own independent setup.

Please note our response (RE7) to reviewer LRnR where we summarize the reasons why we used several different protocols mostly for comparison to existing work. We added a summary of evaluation setups in Section 4.4 to clarify this confusion.

RE5: Explain that “Can.Inv. = Canadian Invertebrates or CanadianInvertebrates-ML” in the caption of Table 3 | CanadianInvertebrates-ML is defined in Line 250 but never used.

We use the CanadianInvertebrates-ML data splits but referred to these experiments simply as Canadian Invertebrates. To avoid confusion, we now only use CanadianInvertebrates-ML consistently throughout the paper, including in the caption of Table 2.

RE6: Why is there no predictive reliability analysis for the arthropods datasets?

The arthropod reliability analysis is included in Appendix E and referenced at line 462. Since it confirms the findings on MycoAI without adding new insights, we placed it in the appendix due to the page limit.

RE7: Section 5 could be structured as Sec 5.1 Taxonomic Classification Accuracy, Sec 5.2 Predictive Reliability, and Sec 5.3 Unknown Species, following Sec 4.2.

We thank the reviewer for this suggestion. Our structure was organized by taxonomic groups rather than by task to highlight gains on the most difficult setup first.

We agree that the proposed structure provides a clearer flow and have adopted it in the revised manuscript (Section 5).

AuthorsOpenReview ↗

Rebuttal (Part 2)

RE8: Which datasets were used to evaluate unknown species detection and placement?

Old experiments used the augmented MycoAI dataset for training only, and the held-out unknown test sequences to report the results. However, this data and experimental setup are replaced with the new unknown species evaluation protocol built entirely from the MycoAI Benchmark (Section 4.1, 5.3) and is explained in our general response. We also state again in Section 5.3 the data splits used.

RE9: Why can’t BarcodeBERT or BarcodeMamba be used for kNN-based unknown-taxa detection?

The reason we did not add these baselines is that k-NN classifiers have no built-in mechanism for detecting unknown taxa. Adding one requires a decision rule, such as a distance threshold learned from data, which goes beyond the k-NN setup used in the original BarcodeBERT and BarcodeMamba works.

However, we added another kNN-calibrated baseline with a rejection threshold (Section 4.3) to report this. Model implementation details are in Appendix H, and its results are added to the unknown species experiment (Section 5.3). These baselines underperform PROTAX-SSL.

ModelDet. @ FPR=5%Det. @ FPR=10%Det. @ FPR=15%Det. @ FPR=20%Placement (%)AUROC
BarcodeBERT (k-NN-calibrated)11.522.432.041.149.869.4
BarcodeMamba (k-NN-calibrated)25.539.148.656.572.773.2
PROTAX-GPU10.121.735.247.149.574.4
PROTAX[BarcodeBERT]12.825.637.949.550.075.2
PROTAX[BarcodeMamba]27.6‾\underline{27.6}43.554.262.573.179.3
PROTAX[BarcodeBERT+Mamba]27.6‾\underline{27.6}44.7‾\underline{44.7}56.3‾\underline{56.3}64.8‾\underline{64.8}73.4‾\underline{73.4}80.4‾\underline{80.4}
PROTAX[BarcodeBERT+Mamba, MLP]30.346.757.665.774.780.8

RE10: Line 50: “but the underlying data remain difficult”. What does it mean?

The sentence means that although DNA barcoding scales identification beyond what morphological methods allow, it does not resolve the fundamental challenges of biodiversity data, which are described immediately after it: arthropods and fungi are among the most diverse and taxonomically complex groups of life, with highly imbalanced species distributions, substantial gaps in species annotations, and incomplete reference libraries.

We have rephrased the sentence to state this more clearly (line: 49).

RE11: Open-set recognition vs unknown species detection

We used “open-set recognition” to connect the unknown-species experiments to the term common in the ML community. We agree that using two terms causes confusion, and we now use “unknown-species detection” consistently throughout the paper.

RE12: Have an additional column in Table 1 to show which models are reference-based and which are not.

We agree this distinction should be visible in the table. Since Table 1 is already wide, we marked non-reference-based models with a symbol (†) next to their names and explained it in the table caption.

RE13: Section 4.3 Baselines: Why are the baselines directly taken from Gao et al. (2025)?

Gao et al. (2025) is, to our knowledge, the only work that evaluates these baselines on all three MycoAI evaluation sets used in our study (the MycoAI Benchmark, Yeast, and Filamentous) under a single consistent setup. BLAST, the only reference-based method among them, was evaluated with self-match exclusion, matching our protocol. The remaining models are not reference-based, so self-match exclusion does not apply to them, and reproducing them would follow exactly the same training procedure and evaluation setup as Gao et al. (2025). As noted in Section 4.3, we treat these models as an indirect comparison with a different kind of state-of-the-art classifier.

Reviewer_LRnROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose tackle the task of taxonomic classification from DNA barcodes by leveraging embeddings from DNA barcode foundation models into PROTAX, a probabilistic hierarchical classification approach. PROTAX originally operates using raw sequence similarity and the authors propose to use similarity in the embedding space of DNA barcode FMs instead.

The authors compare their approach PROTAX-SSL (using different FM embeddings or combination of embeddings ) with the original PROTAX methods and the DNA barcode FMs BarcodeBERT and BarcodeMamba on taxonomic classification of fungi and arthropods (the FMs are pre-trained on arthropod data). They also study the calibration of these models. Additionally, they compare the capabilities of these model to classify unknown species correctly at the genus level on fungi.

The authors find that the method is most advantageous compared to other methods when used on groups of species the FMs have not been trained on, and for detecting unknown species and placing them in the correct genus.

Critical strengths

The paper is generally clear and reads well. It presents a quite straightforward idea, and combining the strengths of FMs with PROTAX is presented as a simple and relevant idea.

The method seems most promising for detecting unknown species, improving significantly from PROTAX, and for providing calibrated predictions.

I found the analysis of representation spaces in appendix F informative, though it was a bit buried and I would have liked to see further discussion, since a point of the authors is that the underlying embedding space is crucial to the success of PROTAX-SSL.

Critical weaknesses
  • the different evaluation setups are bit confusing, in particular the self-match exclusion evaluation even though the authors say this is what seems to have been used in PROTAX. I
  • The "best" version of PROTAX-SSL *( barcodeBert + barcodeMamba + MLP) performs very slightly better or similarly to other methods on the task of species classificationbut takes significantly more time to train, and it it not clear that the compute/performance increase tradeoff does not justifies using this method. Basically the advantage of PROTAX-SSL for species classification are not clearly demonstrated if one looks at accuracy. It does offer advantages in terms of calibration but not consistently across the different datasets.
Questions for authors
  • I am curious why the authors have not used BarcodeMamba+ (Gao et al. 2025)? I wonder whether PROTAX-SSL with this FM would really be advantageous over BarcodeMamba+ given the very minimal difference between BarcodeMamba and the PROTAX-SSL[BarcodeMamba] for example.

  • The performance difference between BarcodeMamba and the PROTAX-SSL[BarcodeMamba] and between BarcodeBERT and the PROTAX-SSL[BarcodeBERT] is not always significant (or sometimes 0.1 accuracy point, and I don't really know how much people using these models would care about in practice), and the variant which seems to be most advantageous (at least on MycoAI) is PROTAX-SSL[[BarcodeBERT+BarcodeMamba]. This could make the reader question whether the advantage is from the combination of the 2 FM representations rather than from PROTAX. I have 2 questions related to this point:

    • why have the authors not included a comparison which combined BarcodeBERT+BarcodeMamba? If not, what is challenging?
    • does the slight increase in performance justify using PROTAX-SSL[[BarcodeBERT+BarcodeMamba] which takes significantly longer to train (as reported in table 5)?
  • I am confused by fact that there is the identical barcode overlap of 86.73% for the Yeast dataset. If I understand correctly, I understand you are using selfmatch exclusion for these samples, but even so how is there no information leak between training and testing?

2 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

We thank the reviewer for the positive assessment and constructive feedback. We are glad they found the paper well-motivated and results promising. We address each concern below.

RE1: Why BarcodeMamba+ is not included in our work:

BarcodeMamba+ is not yet publicly available. Since PROTAX-SSL treats the encoder as a frozen, interchangeable component, we plan to add this variant once it is released.

Meanwhile, our Canadian Invertebrates experiments are closely analogous to this setting. PROTAX[BarcodeMamba+] on MycoAI would pair an encoder with data from its own pretraining domain, and on Canadian Invertebrates the encoders were likewise pretrained on arthropod COI barcodes from the same library. In that setting, PROTAX[BarcodeMamba] preserves BarcodeMamba’s accuracy. We expect similar behaviour with BarcodeMamba+, with the potential of combining its space with other spaces leading to further accuracy improvement on the species level (assuming like BarcodeMamba and BarcodeBERT it makes different errors).

Additionally, our use of arthropod-pretrained encoders on fungal ITS was a deliberate choice. It highlights the calibration gains of PROTAX-SSL when the underlying embedding space is mismatched.

RE2: On the size of the accuracy gains over the underlying encoders:

We agree that the accuracy differences between each encoder and its PROTAX-SSL variant are small. PROTAX-SSL is not meant to compete with the foundation models it builds on, but to make their predictions trustworthy and to extend what they can do, without losing accuracy. Its main benefits lie in:

  • Calibration: PROTAX-SSL corrects the overconfidence of k-NN classifiers across datasets, highlighting which identifications to trust. This matters for downstream ecological monitoring and decision-making.
  • Capabilities simple classifiers lack: PROTAX-SSL adds principled modeling of unknown taxa for open-set recognition, predictions that are consistent with the taxonomic hierarchy, and the inclusion of multiple predictors. k-NN and linear probes have no native mechanisms for these features.
  • Modular design: The PROTAX[BarcodeBERT+Mamba, MLP] variant demonstrates the potential accuracy gains that can be achieved. We believe more deliberate choices of extracted features, combinations of embedding spaces or data modalities, and MLP network design can push this further. Because the framework is modular and encoder-agnostic, the community can readily combine and extend it with new foundation models, features, modalities, and fundamentally different classifiers such as BLAST.

RE3: A comparison combining BarcodeBERT+BarcodeMamba with k-NN:

Our original baselines followed the standard evaluation of each encoder (single-space k-NN). We agree that a combined k-NN baseline is needed to isolate where the multi-encoder gains come from and is added to Appendix G.

BarcodeMamba embeddings occupy a narrow cone (random pairs have a cosine similarity of about 0.98), so naive concatenation is dominated by BarcodeBERT’s wider similarity range. We therefore evaluated both plain concatenation of L2-normalized embeddings (equivalent to averaging the cosine similarities from the two spaces) and concatenation after mean-centering each space. Both were evaluated under the same setup as our MycoAI experiments, including self-match exclusion. The table below shows their performance:

ModelMycoAI BenchmarkYeastFilamentous
BarcodeBERT (k-NN)60.962.936.7
BarcodeMamba (k-NN)77.386.457.4
Concat (k-NN)63.867.839.6
Centered concat (k-NN)72.280.249.3
PROTAX[BarcodeBERT+Mamba]79.0‾\underline{79.0}86.2‾\underline{86.2}57.5‾\underline{57.5}
PROTAX[BarcodeBERT+Mamba, MLP]80.485.758.1

Both BarcodeBERT+Mamba (k-NN) variants are substantially worse than BarcodeMamba (k-NN) alone, showing that simply fusing the embeddings is not the source of the gains of the multi-embedding PROTAX-SSL variants. Instead, PROTAX-SSL learns the reliability of the features extracted from each space and how to combine them. This lets it exploit the complementary information in the two spaces, mostly outperforming each space on its own, which simple downstream classifiers fail to do.

AuthorsOpenReview ↗

Rebuttal (Part 2)

RE4: Does the slight increase in performance justify using PROTAX-SSL[BarcodeBERT+BarcodeMamba]?:

Our results show that no single PROTAX-SSL variant is best across all settings, so the choice should depend on the data. On large, noisy datasets with complex decision boundaries, such as the MycoAI Benchmark, PROTAX[BarcodeBERT+Mamba, MLP] improves species-level accuracy over single-embedding variants. On simpler settings such as the Yeast set, or where raw sequence space already separates taxa well, as in FinBOL, single-embedding variants or PROTAX-GPU are sufficient.

Much of the multi-embedding variants’ longer training time reflects GPU memory limits on batch size rather than model complexity. Since training is a one-time cost, we view the MLP variant as an option for applications with noisy, complex data, where species-level accuracy and calibration matter most.

RE5: Information leak concern between training and testing:

The risk of information leakage is limited by the scale of the data and the low capacity of the classifier. The majority of parameters in PROTAX-SSL are in the barcode encoders, which are always trained on a distinct external dataset and frozen throughout all experiments. The PROTAX-SSL classification layer is trained only on low-dimensional distance features extracted per candidate taxon (two per branch for single-embedding variants), not on the sequences themselves. Its capacity to encode sequence-specific information across hundreds of thousands of training examples is therefore minimal. Appendix D confirms this empirically: on a held-out split of the MycoAI Benchmark, train and test accuracy and calibration are nearly identical for all variants, including the MLP variant. Self-match exclusion is a common evaluation protocol for reference-based taxonomic classifiers, used in the original PROTAX (Somervuo et al., 2016) and FinPROTAX (Roslin et al., 2022), and in related leave-one-out form for other classifiers such as the RDP (Wang et al., 2007).

However, we note that the Yeast set is not intended as a held-out test set. It evaluates performance within a taxonomically concentrated fungal clade. For held-out evaluation, the Filamentous set is nearly independent of the training data (only 6.48% identical-barcode overlap), and Appendix D provides fully held-out MycoAI Benchmark test results. We kept the full MycoAI Benchmark as the primary evaluation because it evaluates every sequence as a query, including rare species, against the most complete reference library.

RE6: Would have liked to see further analysis on representation spaces, like that discussed in appendix F:

We thank the reviewer for this comment. Due to the page limit, we will keep the figures in Appendix F, but we added a short discussion of its findings to Section 5.4, where we discuss when learned representations help.

In brief, embedding spaces benefit PROTAX-SSL when intra- and inter-species distances overlap heavily in raw sequence space, as in MycoAI, where BarcodeMamba separates species far better than p-distances. When raw sequence space already separates species well, as in FinBOL, embeddings are not beneficial. This also gives practitioners a simple diagnostic: comparing these distance distributions on their reference library indicates whether learned representations are likely to help.

RE7: The different evaluation setups are bit confusing:

We agree the evaluation setups need to be presented more clearly. They serve different purposes:

  • Self-match exclusion on the training data (MycoAI Benchmark, FinBOL, CanadianInvertebrates-ML training split) attempts to mimic classifier use in practice and follows the PROTAX protocol: every sequence is evaluated as a query against the full reference library, with its identical copies removed.

  • The held-out MycoAI Benchmark split in Appendix D confirms that self-match exclusion yields the same accuracy and calibration as fully held-out evaluation.

  • The Held-out test sets are the most familiar evaluation in the ML community, and seek to evaluate generalization: the CanadianInvertebrates-ML test split (fully held-out), and the Filamentous set (mostly held-out).

  • Literature baselines are not reference-based, so self-match exclusion does not apply to them, and they serve as an indirect comparison.

We added a subsection to summarize these setups (Section 4.4).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and corrections

We thank the three reviewers for their constructive feedback. The revised manuscript is uploaded; changes are coloured by the corresponding review (legend on the title page).

In this response, we summarize a change to the paper that we want all reviewers to be aware of. Reviewer q7vq raised a valid concern about our unknown-species protocol: sequences annotated only to genus level do not guarantee that they belong to species absent from the reference library. We accept this criticism, and have replaced the original experiment with a more rigorous protocol (Section 4.1):

  1. Withholding species: To simulate novel species, we withhold a subset of species from the MycoAI Benchmark, removing all of their sequences. A species is eligible to be withheld only if its genus contains at least one known species, and only if it has at most 500 sequences, so that a few large species do not dominate the unknown queries. From the eligible species, we randomly withhold 2,156 species, 15% of all species in the dataset. The taxonomy and reference library are built only from the remaining known species, so no withheld species appears in the taxonomy.
  2. Disjoint training and test unknowns: The withheld species are randomly split into two disjoint halves of 1,078 species each. The first half (20,451 sequences) is added to the training data alongside the known species, and the second half (20,631 sequences) is used to test unknown-species detection and placement. No species appears in both halves.
  3. Evaluation: The evaluation set combines known and unknown queries. We report detection rate of the unknown test sequences at multiple false-positive rates (FPR) of known sequences classified as unknown. FPR is computed over all remaining known sequences with self-match exclusion to reflect the full range of known species. We also report AUROC (independent of any FPR threshold), along with the genus-level placement accuracy of the unknown test queries, which is not conditional on detection; the two are reported separately. Additionally, we added a calibrated k-NN model with a rejection threshold as another baseline (Appendix H).

Below are the results reported in Section 5.3:

ModelDet. @ FPR=5%Det. @ FPR=10%Det. @ FPR=15%Det. @ FPR=20%Placement (%)AUROC
BarcodeBERT (k-NN-calibrated)11.522.432.041.149.869.4
BarcodeMamba (k-NN-calibrated)25.539.148.656.572.773.2
PROTAX-GPU10.121.735.247.149.574.4
PROTAX[BarcodeBERT]12.825.637.949.550.075.2
PROTAX[BarcodeMamba]27.6‾\underline{27.6}43.554.262.573.179.3
PROTAX[BarcodeBERT+Mamba]27.6‾\underline{27.6}44.7‾\underline{44.7}56.3‾\underline{56.3}64.8‾\underline{64.8}73.4‾\underline{73.4}80.4‾\underline{80.4}
PROTAX[BarcodeBERT+Mamba, MLP]30.346.757.665.774.780.8

Although results show lower absolute detection rates under this harder protocol, the main conclusions hold: all PROTAX-SSL variants outperform PROTAX-GPU, PROTAX-SSL outperforms the calibrated k-NN built on the same embeddings, and the multi-embedding variants perform best across all metrics. AUROC increases from 74.4 for PROTAX-GPU and 73.2 for the best k-NN-calibrated model to 80.8 for PROTAX[BarcodeBERT+Mamba, MLP].

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.