- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PROTAX-SSL integrates frozen BarcodeBERT and BarcodeMamba representations into the hierarchical classification framework of PROTAX-GPU. Embedding distances supply taxon-level similarity features, with linear and MLP variants supporting single or combined representations. Evaluation covers fungal and arthropod datasets, predictive calibration, and detection and genus-level placement of purported unknown species. Results show substantial gains over raw-sequence PROTAX on challenging fungal data, improved calibration over similarity-based confidence, and smaller or mixed benefits on curated arthropod data.
Critical strengths
The paper addresses a useful gap between strong barcode representations and reliable taxonomic decisions. Combining learned representations with a classifier that explicitly models taxonomic hierarchy and unknown branches provides a practical approach to incomplete reference databases. The fungal results show that this combination can improve accuracy while producing more reliable confidence estimates. The evaluation includes two encoder architectures and both in-domain arthropod and out-of-domain fungal data. The FinBOL results also show where learned representations offer little benefit over raw sequence distances. This comparison helps establish the conditions under which the proposed combination is useful.
Critical weaknesses
The unknown-species evaluation does not establish that its test examples belong to species absent from the reference library. Section 4.1 selects sequences annotated only to genus level and treats them as unknown species. However, missing species annotations do not establish biological novelty: these sequences could belong to species already represented in the library. Selecting different sequences from the same genus also does not establish species separation between the unknown training and test examples. This leaves the reported 99.9% detection accuracy difficult to interpret as evidence of unseen-species detection.
The calibration comparisons do not isolate the benefit of PROTAX from ordinary score calibration. The encoder baselines use cosine similarity as a proxy for probability, while PROTAX learns a supervised mapping from distance features to probabilities. The paper does not compare against validation-calibrated distance-based classifiers or comparable rejection rules. Since the underlying PROTAX framework already supports heterogeneous predictors, stronger controls are needed to establish what the proposed integration adds beyond improved features and a learned confidence mapping.
Questions for authors
-
How was species absence verified for the genus-only sequences used as unknown examples? Can the authors provide an evaluation that withholds fully identified species from both classifier training and the reference library?
-
Does Table 2 measure detection on unknown queries alone or on a mixture of known and unknown queries? Please report the known-species false-positive rate and clarify whether placement accuracy is conditional on successful detection.
-
How does PROTAX-SSL compare with a validation-calibrated classifier using the same embedding distances, including a distance-based unknown-rejection rule? This comparison would help separate the benefits of hierarchical modeling from those of supervised calibration.g the complete taxonomic path.
4 comments
Rebuttal (Part 2)
RE3: How does PROTAX-SSL compare with a validation-calibrated classifier using the same embedding distances, including a distance-based unknown-rejection rule?
To answer this, we implemented a k-NN-calibrated model as an additional baseline. It uses the same embeddings, training data, and supervision as PROTAX-SSL, so any difference between the two can be attributed to the PROTAX framework.
At each taxonomic level, every candidate taxon is scored by its distance to the query, defined by the taxon’s closest reference sequence, which mirrors PROTAX’s minimum-distance feature. These distances are converted into probabilities by a softmax with a learned temperature for that level, and each level is calibrated independently. At the species level, we add an unknown-species option with a learned distance τ. The temperature and τ are fitted by minimizing the negative log-likelihood of the correct answers. All details are in Appendix H.
Calibration curves and ECE of this model are added to Figure 2 and compared with the base k-NN and PROTAX models in Section 5.2. The k-NN-calibrated models improve calibration considerably over the standard ones, but they generally remain behind PROTAX-SSL, which indicates that PROTAX-SSL learns more than a simple mapping from similarity scores to probabilities.
Unknown species detection results are reported in the previous answer (RE2).
RE4: PROTAX-SSL Contribution
PROTAX-SSL learns a probabilistic decision model. It combines distance-based features with features describing the reference taxonomy. The distance features capture more than the query’s proximity to a taxon: carefully designed features such as the gap between the closest and second-closest distances describe how decisive that proximity is. PROTAX-SSL then learns the reliability of each feature, and how different combinations of feature values map to correct predictions at each level. The hierarchical structure adds more. At each node, probability is distributed only among the children of the selected parent, including an unknown branch, rather than across all classes in a taxonomic level as in typical classifiers. Probabilities then propagate down the hierarchy, so uncertainty at a higher level carries into the levels below it.
Finally, while uncertainty estimation is a rich topic in machine learning, it remains largely unaddressed in taxonomic classification with barcode foundation models. To the best of our knowledge, this is the first work to address the uncertainty of barcode foundation models, providing both an analysis of their calibration and a method to improve it. It is also, to our knowledge, the first to combine multiple barcode foundation models, and as shown in our response to reviewer LRnR (RE3) and in Appendix G, simple fusion of their embedding spaces degrades performance.
The results you provided demonstrate an improvement in the ability to detect unknown species, but I am still concerned about the trade-off between computational cost and performance. Specifically, given the modest AUROC improvement from 80.4 to 80.8 when moving from the non-MLP dual-embedding variant to the MLP variant, is the additional training cost justified for such a small gain?
We thank the reviewer for the follow-up. We agree that the AUROC gain of the MLP variant is small. AUROC aggregates detection over the entire range of FPR thresholds, so differences at individual thresholds are diluted in the overall score. At fixed FPRs of 5% and 10%, the MLP variant shows larger gains in detection. Independently of any detection threshold, it also achieves the highest genus-level placement accuracy for unknown species. Its gains are also larger in species-level classification accuracy in the MycoAI experiments (Section 5.1) and in calibration, most notably at the species level (Section 5.2).
As our results show, no single PROTAX-SSL variant performs best in all cases, so the value of the trade-off between cost and performance depends on the use case. We therefore recommend the MLP variant when unknown species must be detected at low false-positive rates or placed in their correct genus, and for applications where species-level accuracy and calibrated uncertainty matter most.
Regarding cost, training is a one-time expense, and as explained in our response to Reviewer LRnR (RE4), much of the multi-embedding variants’ longer training time reflects GPU memory limits on batch size rather than model complexity. At inference, the linear and MLP dual-embedding variants run in almost the same time, since they share the encoders and distance computations.
Finally, the PROTAX[BarcodeBERT+Mamba, MLP] variant demonstrates the potential of combining complementary sources of evidence, and shows that an MLP can exploit them more effectively than a linear layer. We expect this advantage to become more pronounced with more deliberate choices of extracted features, embedding spaces or data modalities.
Rebuttal (Part 1)
We thank the reviewer for the careful and constructive review. Following the reviewer’s concerns, we have replaced the unknown-species experiment with a more rigorous protocol and added a calibrated k-NN baseline with a rejection threshold, as summarized in our general response. We address each point below.
RE1: Unknown Species Evaluation Setup
We thank the reviewer for this important point. We agree that genus-only annotations do not guarantee that a sequence belongs to a species absent from the reference library. Since our problem statement and motivation center on detecting novel species, we have replaced this experiment with a setup in which unknown species are simulated by withholding whole species from the training data and the reference library.
Please refer to our general response above for the experiment details and results. The new protocol is added to Sections 4.1 and 5.3 in the revised manuscript.
RE2: New Unknown Species Evaluation Results:
We train the PROTAX models and the k-NN-calibrated models (described in our next response and Appendix H) on the unknown species training split. False-positive rates (FPRs) for unknown species detection are measured on the known queries, evaluated with self-match exclusion: for a given FPR, the detection threshold is set so that this fraction of known queries is wrongly flagged as unknown. As shown in Appendix D, self-match exclusion yields the same results as fully held-out evaluation.
Detection rates in Table 3 are reported on the unknown species test sequences only, all of which belong to species absent from both training and the reference library. Genus Placement accuracy is reported on the same sequences and is independent of detection, so it does not depend on the chosen FPR. AUROC, computed over both known and unknown queries, likewise summarizes detection independently of any FPR threshold. ROC curves are reported in Appendix I.
Table 3 results now show lower absolute detection rates when compared to the original experiment, reflecting a more realistic and generalized assessment of unknown species detection under this robust protocol. However, our main conclusions still hold: PROTAX-SSL variants outperform PROTAX-GPU, PROTAX-SSL outperforms the k-NN-calibrated + rejection models built on the same embeddings, and the multi-embedding variants perform best across all metrics. AUROC increases from 74.4 for PROTAX-GPU and 73.2 for the best k-NN-calibrated model to 80.8 for PROTAX[BarcodeBERT+Mamba, MLP].