- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether functional similarity to training data can help identify reliable microbial trait predictions under taxonomic shift. Using a fixed ESM2-based logistic classifier, it compares coverage scores based on protein embeddings, taxonomy, and KEGG pathways. It also combines coverage with classifier confidence for selective prediction. The study audits 15,002 accessions, reconstructs label and split provenance, and regenerates predictions. KEGG coverage ranks prediction correctness better on three focal proxy traits, but family-bootstrap intervals leave its selective-prediction benefit uncertain. The authors present these findings as an exploratory diagnostic and data audit, with appropriately limited claims about reliability.
Critical strengths
-
The data audit is useful in its own right. In particular, it shows that negative pathogenicity labels come from a biosafety-level fallback rather than explicit negative annotations. This changes how the focal results should be interpreted and illustrates a problem that can be missed when evaluating biological prediction models.
-
The authors separate prediction quality, correctness ranking, and selective utility instead of treating them as interchangeable. They report family-bootstrap uncertainty, undefined resamples, and the stronger performance of ordinary confidence on the other ten traits. This makes the paper informative even where the proposed score does not improve selection.
Critical weaknesses
-
The three focal targets provide less independent evidence than their number suggests. All 7,769 human and 8,008 animal negative labels come from the BSL-1 fallback, coupling the pathogenicity and biosafety targets (Section 3). Appendix A reports additional correctness-ranking results for traits such as Gram stain and catalase. These are useful, but they do not resolve the interpretation of the focal selection analysis.
-
It is difficult to tell what part of the KEGG signal is specifically functional. The taxonomy baseline uses a coarse shared-rank score (Section 4.1). Finer lineage structure, class prevalence, trait difficulty, or annotation source could also affect correctness ranking. The current comparison leaves these explanations open, which limits confidence that the signal will transfer to another dataset.
-
The comparison with existing reliability scores is incomplete. Section 2 acknowledges that class-conditioned Trust Score was not tested. Bacformer is a useful coordinate control and outperforms KEGG on its matched subset, but it answers a different question. A class-conditioned neighborhood score would help establish whether functional coverage adds information beyond a stronger confidence baseline.
Questions for authors
-
Which non-focal traits use direct annotations, and which depend on fallback or derived-label rules? Appendix A reports KEGG correctness AUROCs of 0.726 for Gram stain and 0.563 for catalase. Clarifying their provenance would help assess how much independent biological support these results provide.
-
Using the existing predictions, does the KEGG signal persist within comparable taxonomic groups or annotation sources after accounting for class support? Please indicate where the data are too sparse to support this analysis.
-
How does KEGG coverage compare with a class-conditioned neighborhood score such as Trust Score, using the same fitted classifier and held-out families?