acceptodds
Under review as a conference paper at ICLR 2027

FUNCTIONAL COVERAGE FOR SELECTIVE MICROBIAL TRAIT PREDICTION UNDER TAXONOMIC SHIFT

Abstract

Functional-pathway similarity may help identify reliable predictions for microbial lineages absent from a model's labelled training set. We evaluate this hypothesis on 15,002 genome accessions using taxonomic holdouts, a fixed protein-embedding classifier, and three complementary coverage coordinates. In the initial family-held-out analysis, KEGG proximity ranks correct predictions above errors on three pathogenicity/biosafety targets with mean AUROC 0.652, compared with 0.531 for protein-embedding proximity. We then examine whether this diagnostic signal improves selective prediction, regenerating accession-level predictions and quantifying uncertainty across families. Combining pathway coverage with classifier confidence yields a mean balanced-accuracy advantage of 0.069 over confidence alone at 50% retention, but the paired family-bootstrap 95% interval [-0.055, 0.202] includes zero. Data-provenance checks further show that all negative pathogenicity labels are biosafety-derived proxies and that a small number of records have inconsistent or placeholder taxonomy. Across 468 refitted taxonomic-holdout models, aggregate phenotype AUROCs reproduce to within 0.00074. Our results distinguish informative coverage geometry from established selection utility: functional proximity is a useful exploratory diagnostic in this setting, but a robust reliability improvement is not demonstrated. We provide prediction-level supplementary results and reproducible statistical checks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
microbial phenotype prediction, out-of-distribution generalization, taxonomic distribution shift, selective prediction, functional coverage, protein language models, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ofr7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether functional similarity to training data can help identify reliable microbial trait predictions under taxonomic shift. Using a fixed ESM2-based logistic classifier, it compares coverage scores based on protein embeddings, taxonomy, and KEGG pathways. It also combines coverage with classifier confidence for selective prediction. The study audits 15,002 accessions, reconstructs label and split provenance, and regenerates predictions. KEGG coverage ranks prediction correctness better on three focal proxy traits, but family-bootstrap intervals leave its selective-prediction benefit uncertain. The authors present these findings as an exploratory diagnostic and data audit, with appropriately limited claims about reliability.

Critical strengths
  1. The data audit is useful in its own right. In particular, it shows that negative pathogenicity labels come from a biosafety-level fallback rather than explicit negative annotations. This changes how the focal results should be interpreted and illustrates a problem that can be missed when evaluating biological prediction models.

  2. The authors separate prediction quality, correctness ranking, and selective utility instead of treating them as interchangeable. They report family-bootstrap uncertainty, undefined resamples, and the stronger performance of ordinary confidence on the other ten traits. This makes the paper informative even where the proposed score does not improve selection.

Critical weaknesses
  1. The three focal targets provide less independent evidence than their number suggests. All 7,769 human and 8,008 animal negative labels come from the BSL-1 fallback, coupling the pathogenicity and biosafety targets (Section 3). Appendix A reports additional correctness-ranking results for traits such as Gram stain and catalase. These are useful, but they do not resolve the interpretation of the focal selection analysis.

  2. It is difficult to tell what part of the KEGG signal is specifically functional. The taxonomy baseline uses a coarse shared-rank score (Section 4.1). Finer lineage structure, class prevalence, trait difficulty, or annotation source could also affect correctness ranking. The current comparison leaves these explanations open, which limits confidence that the signal will transfer to another dataset.

  3. The comparison with existing reliability scores is incomplete. Section 2 acknowledges that class-conditioned Trust Score was not tested. Bacformer is a useful coordinate control and outperforms KEGG on its matched subset, but it answers a different question. A class-conditioned neighborhood score would help establish whether functional coverage adds information beyond a stronger confidence baseline.

Questions for authors
  1. Which non-focal traits use direct annotations, and which depend on fallback or derived-label rules? Appendix A reports KEGG correctness AUROCs of 0.726 for Gram stain and 0.563 for catalase. Clarifying their provenance would help assess how much independent biological support these results provide.

  2. Using the existing predictions, does the KEGG signal persist within comparable taxonomic groups or annotation sources after accounting for class support? Please indicate where the data are too sparse to support this analysis.

  3. How does KEGG coverage compare with a class-conditioned neighborhood score such as Trust Score, using the same fitted classifier and held-out families?

Reviewer_dmk6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits whether KEGG pathway proximity to the training set predicts when a fixed ESM-2 phenotype classifier is correct. On 15,002 genomes, coverage AUROC is 0.652 on three pathogenicity and biosafety targets. The selective-prediction gain of 0.069 has a family-bootstrap interval including zero. All negative labels are BSL-1 proxies.

Critical strengths

The paper reports that every negative pathogenicity label is a BSL-1 fallback and that the three focal targets are coupled by construction, which undercuts its own headline result and is useful to anyone using this benchmark.

Critical weaknesses

All three focal targets use the BSL-1 fallback as their negative class (Sec. 3), so they are near-duplicates of one label, and the bootstrap uses the same family draws across them. The mean of 0.652 and the three per-trait intervals are not three independent results.

Phylogenetic confounding is acknowledged in a figure caption but never tested.

The taxonomic baseline takes only four values (3, 2, 1, 0 for shared order, class, phylum, or none) while KEGG coverage is a continuous cosine, and ties are not averaged. The gap of 0.652 versus 0.548 is partly attributable to score granularity.

The selection advantage of 0.069 has interval [-0.055, 0.202], and all three per-trait intervals include zero, with the biosafety point estimate at approximately zero (Figure 4). The abstract and Section 5.3 still lead with the point estimate.

Questions for authors

Please test whether KEGG coverage ranks correctness after conditioning on phylogenetic distance to the fitting set, for example by stratifying test genomes by minimum patristic or 16S distance.

Please replace the four-valued taxonomic coverage with a continuous taxonomic or phylogenetic distance and average ties, then report the comparison against KEGG coverage.

Please add class-conditioned Trust Score and at least one calibration baseline such as temperature scaling, since the paper states these stronger error-ranking baselines are untested.

Reviewer_dZzsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates whehter a similarity across the functional-pathway space to a model's training data, can identify which microbial phenotype predictions are trustworthy under taxonomic shift. The authors used ~15K genomic accessions, a logistic classifier, and three coverage metrics (embedding proximity, taxonomy [coarse], KEGG-module completeness), reproduced a prior taxonomic holdout ladder, and tested coverage in two ways. KEGG proximity ranks prediction correctness better than the embedding or taxonomy on three focal targets, but when coverage is combined with model confidence for selective abstention at 50% retention, the gain is not significant. In addition, a provenance audit found that every negative pathogenicity label is a BSL-1 fallback rather than an explicit annotation. The authors conclude that functional coverage is informative in an exploratory way, but that there's not robust reliability improvement.

Critical strengths
  1. The paper replaces row-level bootstrap with paired family-level resampling, which is the correct estimator for phylogenetically structured data.
  2. The label-provenance audit is unusual and valuable.
  3. Reproduction is thorough: 468 refits, pinned versions, and historical and regenerated numbers retained.
  4. The claims are calibrated to the evidence- a null is reported as a null, and no overselling of the signal that survives.
Critical weaknesses
  1. The taxonomic baseline is a very weak strawman: 4 values (shared order/class/phylum/none) against 570 KEGG modules, where KEGG module completeness is strongly phylogenetically structured. This means that KEGG proximity has a very fine-grain phylogenetic structure, and should be therefore compared to phylogenetic distance rather than this strawman. Suggestions— GTDB tree distance, AAI, etc.
  2. KOfam-derived completeness also encodes annotation density, which correlates both with training-set membership and with BSL assignment, since clinical isolates are over-representend. Coverage may be reading “well-characterised clade” rather than pathway content.
  3. The study is underpowered for its central question. The interval [-0.055, 0.202] neither supports nor refutes the original claim.
Questions for authors
  1. Can the correctness-ranking comparison be rerun with a continuous phylogenetic distance like GTDB tree distance in place of the four-value taxonomic coordinate? This is feasible with the cached cohort and would determine if the KEGG advantage is functional or phylogenetic.
  2. Does the KEGG coverage signal survive controlling for annotation density, for example by regressing out the number of annotated KOs per genome?
  3. Coverage-gated abstention will preferentially abstain on lineages least represented in reference databases, which in environmental or novel-lineage settings are the organisms of interest. Have you measured the clade composition of the retained versus rejected sets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.